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For some time, the ARI°F~brt Knox Field Unit has been engaged in the de- / 
velopment and application of innovative, effective techniques designed to iri- / 
crease the quality of the products of Army Centers/Schools. These products 
include trained soldiers and training materials. The training evaluation arid 
feedback team of this unit performs research and development on increasing the 
quality of these products by improving the information flow between training 
developers arid evaluators and units in the field. ~K 

To improve the quality of their products, training developers need in- 
formation about the produces from users in the field. Presently much of the 
information available to training developers ori their products is collected* in 
?^ e _^??^_^?_ t F a ^ n ?- n S_^Y a ^ u ^ toi: s. Training developers, however, have indicated 
that the feedback that they receive from the field via training evaluator^ 
does not satisfy all of their feedback nefeds. 

< ■ «_ _\_ _ «_ * 1 ' 

This report examines the types of feedback presently collected by training 

evaluators and, developers at one large Center/School and identifies theif addi- 
tional feedback heeds. _. Present feedback. sources are examined' to determine 
their accuracy ^rid usefulness . The results Suggest that the feedback presently 
available to training developers is lacking in both specificity and objectivity-, 
and thus may be neither accurate hbr useful. Recommendations, for improving 
the accuracy and. usefulness b£ the feedback are given, which haVe implications 
for TRADOd and other Army personnel concerned with bbtairiiri^a«urate , usefjul!* 
feedback from the field. /: 



FEEDBACK NEKDS'OP TRAINING DE 



EXECUTIVE SUMMARY 



V ELOPERS AND L: VALUATORS 



Requirement: 

In order to improve the quality of their products draining developers 
need accurate, detailed feedback from the field. For the most part^ training* 
developers depend .upon Directorates of Evaluation and Standardization (DOES) 
to supply the feedback needed). Recently, training developer! have indicated 
that the feedback that they are receiving ,f rom DOES evalu&tors is not meeting 
all thgir needs. This report documents the kinds of feedback presently avail- 
able to£ training developers jbhrough. training evaluators or other sources, and 
suggests ways to improve the/ flow of useful feedback from' the field to training 
evaluators and developers at/ Centers/Schools. 

Procedure** 

Thirty division and branch chiefs from, the Directorate of Training De- 
velopments (DTD) ,and the Defes at a large Center/School were interviewed con- 
cerning their feedback needs using an interview form specially developed for 
the purpose. After being fold the purpose. of the^ interview, interviewees were 
queried about the types of / feedback currently available to them; For each 
type of feedback mentioned, the interviewees were asked to teii; where they get 
-it, how they- use it, and Where they send it. Interviewees were then asked to 
state any. needs that they/ might have for additional feedback. Results of the 
interviews were recorded on the interview form by the authors. Results ob- 
tained from different interviewees were . integrated for drawing conclusions and 
making recommendations. 



Findings: 

^ The present feedback system is not providing all the feedback that train- 
ing, developers need for/ improving training programs and materials. In the 
case, of training materials, more feedback is needed on their availability and 
use. £ri the case of training programs , considerable feedback is currently 
available, but the feedback tends to lack specificity and is largely derived 
from :subjec|.ive sources^ To increase the quantity and> qudlity of feedback 
available ,^o ^ rain ing^ developers, the coordination between DTD arid DOES per- 
sonnel shotild be increaleS, DTD personnel should increase the number * of con- 
tacts with the field ^ more* emphasis should be placed on hands-on testing and 
other obj ect ive data, collection methods , *#nd computer-based data analysis and 
management techniques for handling feedback should be- developed. 

Utilization of Findings: - 

The- findings of [hi s ^report should be useful to any grou^ or agency in- 
terested in obtaining accurate, feedback on the quality of their products.. 
Training developers and evaluators should find the recommendations included ih~ 
the report particularly useful for improving^ the quality of the feedback that 
they .obtain. ^ 
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Thi- iiifjftv.e of -this p;if)cr is to identity the tnloTnat;tpi^ feedback needs _ 
o! tr.«i:u:i}; <|evelopers and eva Ida tors as the basis -for developing an integrated 
feeOh.uk system' for .feedback o f t n f o ma t ion from centers, schools and opera- 
tional units .to tvaihlng des?el ape and e valuators . PrcsTdnt sourc.es of feedback 
and fredlaek heeds are Identified through •interviews wipir t ra in ing dcyelopers 
.Kid ^valuators at one Center/Sc^ol . Since the reVuilts >re based primarily upon 
a stud: of a single henteV/Schbd il , generalization to, other Centers /Schools 
shonid* be c^ductcd cautiously. 'Mo^t a if itHS : cbtlc las ions . re^h'^d in this paper 
sire general ones 'which should apply ;o other Centers/Schools. Because they^are; 
involVe^'in all phases of trainingVdeVelop^ent, DTD petsonneL provided the data 
on the feedback needs of traiHing developers DTD personnel are involved In tfte 
"performance of the following t ra inirig. development functions: , front-end analysts 
de^lo-^.ent '•of individual and' colT'ec t-ive training materials, devices, and pro- 
^ r; ^s^aad preparation of camera-ready copies of and d i st r ibut ion of institu- 
tional and extension tr^ in ing^ma ter Ul s . ' It was f elt t ha t, based upon their 
experiences, DTD could adequately egress feedback problems -and^ needs from th*, 
rfVH'st? developer's perspective. Therefore, other training developers such as 
'those in the instructional departments/were not interviewed. The DOES _ as t*\e 
.r^irv evaluate? and provider b,f feedba;cjc at the Ce nt e v^c hoc I - elected for 

htnteter data on the feedback sou'rees and needs of t ra in ing e valuator s ; In 
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addition to gathering feedback on personnel, logistics, and training problems 
from National Guard, Reserve, and active units in' the field, the DOES collects 
data on the effectiveness of courses conducted at the Center/School. Other 
functions performed. by the DOES "include providing and coordinating assistance 
to units in the field and coordinating modernization arid deployment of new 
weapons Systems. Although these functions are as much a part of the DOES 1 s 
mission as evaluation, this report will emphasize the evaluative function 
since much of the feedback collected by the DOES is collected for evaluation 
purpo ses. 

* — 

This paper concentrates on the feedback needs as they apply to the products 
of TRADOC Centers/Schools to include trained soldiers, training materials, and 
to a limited extent, training programs. The paper is comprised of five sections , 
Following this section , Section II describes the methods that were used to iden- 
tify the feedback needs of training developers (DTD's) and training evaluators 
(DOES's). Section III describes the collection and use of feedback data by 
evaluators at brie l)OES and lists feedback needs as_perce B ived by the DOES divis- 
ion and branch chiefs . Section IV descr ibes informat ion feedback available tb_ a 
representative DTD from ^several sources , including the DOES, and tells how each 
type of feedback is used by the DTD, with special emphasis on DTD's feedback 
needs . Based on the feedback needs as stated by training developers and evalua- 
tors and on previous analyses of feedback sources and methods (Burns ide, 1981, 
1982), the final section presents conclusions ^about the feedback methods used in 
gathering, analyzing and distributing feedback information on Center/School 
training developmen t and administration procedures . 

While this paper deals primarily with feedback on soldier performance and 
the use of training materials, many of the problems and issues identified apply 
equally to other types of feedback needed by various agencies. Other types of 
feedback needed might include feedback on the utilization of training resources, 
feedback pertinent to the assignment and retention of qualified personnel, and 
feedback needed for managing facilities -and material . The recommendations in- 
cluded in Section V may be used by. any group or agency interested in obtaining 
accurate continuing information on the quality of their products. Training 
developers and evaluators should find the recommendations especially helpful . 
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SECTION II 
INTERVIEW METHODS*" 

Interviewees 




The .personnel interviewed were division and branch chief s from the DTD and 
DOES of a major ^KADOe Center/School^ Each division or branch chief interviewed 
was responsible for management of a major functional area, as outlined below. A 
total of 30 division and branch interviews were conducted early in CY 1982, 
eight in the DOES and ^2 in the DTD. In some cases, experienced, personnel in a 
branch were interviewed jointly with or in addition to the branch chief, partic- 
ularly when the branch chief was relatively new and inexperienced in the posi- 
tion. In the DOES, three division chiefs arid five branch chiefs were inter- 
viewed, arid iri t the DTD six division chiefs and 16 branch chief s_were interviewed. 
These personnel represented all the available division and branch chiefs occupy- 
ing permanent operational positions; chiefs of special teams or administrative 
offices were not included in the "interview sample. Eighteen of the. 30" division/ 
branch chiefs interviewed were military officers, and the remainder were civil- 
ian personnel. The experience of the interviewees in their duty positions ranged 
from a few months to seven years. While no two DTD's or DOES's are organized 
exactly the same, the personnel interviewed are representative of those in any 
TRADOC Center/School DTD and tiOES . The functional areas addressed in the DOES 
interview sample included evaluation of training at the Center/School , assess- 
ment of personnel, logistics, and training matters in active Aafmy, Reserve, and 
National Guard units, and systems analysis . In the DTD the functional^ areas 
addressed included front-end analysis, development of individual and collective 
training materials, programs, and devices, production of training materials, and 
distribution and management of extension training materials . 

„/ 

Interview Form 

/' In order to insure that a consistent structure was followed throughout the 

interviews, a simple form was developed and followed. This form was modified 
based upon experiences during early interviews, and the final version, consis- 
ting of three worksheets, is at Appendix A. The first worksheet contains an 
outline of the introduction to the interview and a general inquiry concerning 
the types of feedback currently available. The second worksheet contains spe- 
cific questions about the feedback identified on the first worksheet, and the 
final worksheet contains a general inquiry about the types of feedback needed . 
that are not currently available. Blank notebook paper was used to record any 
information obtained during the interviews that did not readily fit on one of 
the wo rksheets . 

Interview Procedures ^ 

The two interviewers for this project were the authors of the present re- 
port. Both these individuals had experience conducting similar interviews in 
previous research projects, and they practiced interview administration together 
to insure that they followed the same interview procedures. Each interview was 
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conducted in the interviewee's work area or a convenient confq|ence f ac ii ity ; 
The interviewer recorded the interviewee f ts responses on the appropriate work- 
sheets . ; Art introduction to the purpose and procedures of each interview was^ 
given following the outline on Worksheet 1. Each "interviewee ther^listed the 
major types of feedback available to him or heir, and these, were recorded on 
this worksheet. He/she then answered the questions on Worksheet 2 for each 
type of feedback listed bri Worksheet 1 . The interviewer completed separate 
copies of Worksheet 2 for each type of feedback. The major types of feedback 
needed but not available were then recorded on Worksheet 3, along with any 
general comments received. In some cases a return visit or telephone cati 
was made to ah interviewee to clarify information obtained. Entries on inter- 
view worksheets were revised as necessary after the interview to insure that 
they contained complete 'and readable information. 



SECTION 111 

DOES^ FEEDBACK SOURCES AND NEEDS 

As one of their missions, DOES 1 s provide feedback to Centers/Schools cori- 
cerriing problems existing at training institutions and in the field. DOES^s 
iise a variety of methods to gather the necessary ' information and disseminate 
it to the appropriate .directorates/departments at the Centers/Schools. The 
methods employe! by DOES in collecting and disseminating the feedback informa- 
tion largely determine the quantity an<l quality • of feedback available^ to 
Centers /Schools. * 

Tnts section describes the kinds of feedbac\ provided to the directorates/ 
departments at one Center/School by the DOES at that institution. For each 
type of feedback listed, methods Used by the DOES to obtain and distribute that 
information will be identified and discussed. As a matter of convenience A 
information obtained from units in the field and information collected at the 
institution will be discussed separately. Following the discussion of types\ of 
feedback and feedback methods, the techniques used by the DtfES in reducing arid 
analyzing feedback information are discussed. This is followed by a general \ 
discussion of the Usefulness and reliability of the 'feedback presently provided 
by the DOES . The section concludes with a description of-^the feedback needs of\ 
"The DOEST""- — : _ ■■ 



Although the data presented in this sect ion are based on the study of a 
single TOES, the types of feedback discussed and many of the problems associa- 
ted with the collection and dissemination of this information are common to 
other DOES' s and thus- may provide valuable lessons for DOES' s in general. This 
section^ may also be of use to departments and other directorates for under- 
standing what types of feedback DOES ' s obtain arid how they obtain them. 

Feedback From The Field 

DOES's generally send evaluation teams to units in the 'field to obtain 
information about problems the units might be experiencing. The DOES in this 
study sends teams consisting of two to five members to each, active and reserve 
unit once every two or three years . The .members of. the team typically include 
individuals knowledgeable in the areas df personnel, logistics and training. 
During each visit the evaluation team adWriisters various surveys, question- 
naires, and informal ^interviews to selected unit personnel. Neither ♦systematic 
observation nor formal testing is used in gathering the information. Surveys 
administered by DOES personnel to graduates of the training institution and 
their supervisors require the graduates/ supervisors to rate the graduate's per- 
formance on specific tasks. Questionnaires in the areas of logistics, personnel, 
and training are either sent , to the .unit just prior to a visit or are adminis- 
tered on the spot by DOES personnel S The questionnaires include questions* on 
availability and quality of personnel, equipment, and training materials and are 
generally completed by the ranking officers in the unit. The interviews, most of 
which are also conducted with these officers, address many of the same topics as 
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the qijest ionhaires i but include additional que^ions bTt-^ training materials arid 
f soldier performance and a few fbllow*up questions based bri the unit's responses 
to the questionnaires. 

Ratings of graduate performance* For each graduate -producing course taught 
at the Center/School, graduate supervisor's surveys are completed by supervisors 
of recent graduates. These surveys require supervisors' to rate on a five-point 
scale the performance of each graduate relative to _that of the average performer 
ori specific tasks. The supervisor is also required to rate , on a five-point 
scale, the criticality of each task arid estimate how frequently . the graduate 
being evaluated performs the task. Similar surveys, referred to as graduate 
surveys, *are completed by the graduates themselves, but only by graduates of NCO 
and officer courses. In these graduate surveys, the graduates are required to 
r^te their own performance on each task, estimate »their^ frequency of performance, 
arid judge the criticality of the tasks,. WherN the responses to these surveys in- 
dicate that th* graduate performs certain taslfs poorly, a follow-up interview may 
be conducted to determine what parts of the tasks the graduates can't do and why 
they can't do them. Separate follow-up interviews are conducted for tasks per- 
formed marginally and tasks per formed especially well. One interviewee,- how- 
ever, indicated that follow-up interviews of survey responses were seldom con- 
ducted* in recent visits. The results of supervisor and graduate surveys are 
rarely- reported by the DOES, and it was not clear how or if these results were 
being used . ..J: . 



DOES personnel have indicated that graduate supervisor' s surveys are used 
in lieu of hands-on performance testing because the demands of the latter are . 
extensive in terms of time and resources. However, Burns ide (1982) has suggested 
that subjective appraisals of performance on specific tasks, like those required 
by the DOES surveys, may. not be accurate, and should always be checked against 
'obj ective measures, such as observations or tests of hands-on performance, to 
ensure their accuracy. Discussion with DOES personnel indicates tfiat no a v ccuracy^ 
checks havi been made for the graduate and graduate supervisor's surveys. 

Responses to quest ionnaires. The evaluation team administers three differ- 
ent quest ionnairelT during f ieidrivisits : (1) a personnel questionnaire, (2) a 
logistics questionnaire, and (3j an Army Training and Evaluation Program (ARTEP) 
.questionnaire The personnel questionnaire is generally administered directly to 
v'the battalion commander by the* chief of the personnel division or his representa- 
tive during the visit. Logistics and ARTEP questionnaires, on the other hand are 
mailed to the unit just prior to the evaluat ion team's visit and a member of the 
team collects the completed questionnaire shortly- after arrival in the unit . The 
logistics questionnaire is completed by the battalion logistics officer, battalion 
maintenance officer, and/or executive officer while the ARTEP questionnaire is 
usually completed by .the battalion operations and training officer. The personnel : 
questionnaire includes questions bri the numbers, experience", and training of the ", 
personnel in tie unit. The logistics questionnaire includes additional questions 
on the number df personnel iri the unit having desired educational and experiential 
qualifications. \ Questions regarding the condition and availability of equipment 
needed by the unVt al so appear on the logist ics questionnaire 's do questions ; , 
concerning the ability of unit personnel to demonstrate- proficiency or understanding. 

J 
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in general f uric tlbrial areas (e .g. troubleshooting, use of publications) . _The 
ARTEP questionnaire is used to evaluate the ARTEP with emphasis on the ARTEP 
document . ; Included are *quest ioris designed" to assess the unit's opinions about 
the usefulness of: the ART§P' arid to determine how the unit uses the document. 

While some questions on the .personnel, logistics, and ARTEP questionnaires 
may provide objective, quantifiable data, overall the questionnaires do not 
provide the data in enough detail ' to satisfy tlie needs of training developers. 
As presently stated, many of the ^questions identify general problems but do riot 
pinpoint the source of those problems so that the* problems can be eliminated. 
In order to satisfy the feedback .requirements of training developers, some of 
the questions included in .the personnel, logistics and ARTEP* questionnaires may 
need to be redesigned arid additional questions developed. 

--- - "Informal interviews : Another type of information, gathered during field 
visits consists of data .from informal interviews with battalion and company 
Commanders- and their staffs. Generally the interview covers personnel train- 
ing, and logistics topics.. The usefulness of training materials (e.g. , ARTEPs, 
Skill 'Qualification Tests (SQTs>, Field Manuals,. Technical Manuals, Commanders 
Guides) are typically addressed during the interview, as are some general ques- 
. t ions about soldier performance . Informal interviews are conducted by one or 
more .members of the evaluation team. At least two members of the team indicated 
that they, use a topical outline as a checklist in conducting these interviews. 
-The da^t a collected -dur-ing- the interview -are -recorded- as- long— hand— nat-es -by "the— 
interviewer. - % • I ; 

The DOES surveyed in this stydy relies hieavily on the information collected 
during interviews for identifying problems experienced by units in the field. 
While unit- personnel respond favorably to interviews, preferring them to ques- 
tionnaires (Burnside, 1981), lack of structure in many interyiew situations may 
result in a failure to capture much important data arid hinder comparison of data 
across units. The interviews. may need to be structured by including a standard 
^set of questions' *tlmt are askjed of every unit visited. As the reliance on the 
interview as one's primary source of- data increases, the* importance of structur- 
ing the interview also increases. .... 
/ 

■ * „ .■ 3? 

Trip Reports . The' DOES uses the information gathered during visits to the 
field as input to trip reports written following each visit. These reports aye 
usually written: by the evaluation team leader. Each trip report consists of a 
series of statements about the unit visited* Many of the statements identify 
problems that the unit is ^experiencing . In the trip reports, personnel, logis- 
tical, and training problems are generally listed separately. Typical problems: 
lifted in trip reports include the unavailability of certain 'training materials 
and equipment or unit dissatisfaction with the materials and equipment that are 
available. Among other problems frequently mentioned . are shortcomings in insti- 
tutional training programs or perceived deficiencies in the performance of the 
graduates of those programs; Personnel 1 shortages of personnel having skills 
critical: to the functioning of the unit is another frequently mentioned problem. 
Although* any problem identified during a 1 field visit might be included in a 
trip repott^ one team leader for the evaluation team stated that frequency of 



occurrence arid problem . criticaiity are prime considerations in deciding which 
problems to include. However, no running tally is kept on the frequency of 
occurrence of different problems, and the re; are rib set criteria for judging 

either the frequency or criticaiity of a. problem. Selection of problems; for • 

inclusion in trip reports then may* best be described as* based on the subjective 
judgment of the team leader. / 

Trip report extracts . Once assembled, the complete trip report with 
executive summary arid iricldsures is provided to the Commanding General of the 
Center/ School for his review. The Commanding General routinely uses information 
from trip reports in his monthly letter to commanders in the field. Upon 
approval -of the Commanding General, applicable portions of trip reports are ex-* 
tracted and transmitted to the Center/School departments/directorates and to 
ma jo r- -commands (TRADOG ,- FORSCOM s 0AR€^-,- --Te^ma^ntai^^he anonymity the- 
-„ Jt _ :"«5-„:izj- : j *. ~^ r*y-~ .nr^T-TTiai i ir flplptpH frrim the trio report 



ma jo re ommanu & ~\ ^jsaiaj^ — r^r^-v^^f t%- yntxwr^ — *« _ - — r.j - - 

units evaluated, unit designations are -normally deleted from the trip report 
extracts. The' trip report extracts are the primary means by which the DOES 
informs the Center/School departments/directorates and major commands about _ 
problems experienced by units in. the field. Trip report extracts 'usually include 
a request that the DOES be informed of any actions- that the department/directorate 
takes to address the problems identified. * , 

Uni.t training materials. Units surveyed during field visits often voluntar- 
ily provide training materials that they have developed for their own use. Upon 
t he-rr -return" to' the een tex/ School- r DOES -eva-lua t ion team personnel place these 
materials in a file. If the materials are part icularly^good , or if they are 
likely to be of general interest - 9 they may be added to the trip report as 

Inclosures. v n 

« - 
Oth er field feedback media . Besides providing input for trip reports, the 
information obtained from the surveys and questionnaires completed during field 
visits provides feedback in other ways. For example, completed ARTEP question- 
naires are. sent to the DTD, where they are available as feedback for updating 
or modifying ARTEP documentation. The DDES personnel division compiles quarter- ^ 
ly summaries from the responses to the personnel questionnaire. These summaries 
present totals, averages, or other descriptive statistics in tabular form for 
each questionnaire item across the units surveyed during that quarter. Quarter- 
ly summaries are kept on file at the DOES and are sent to the Directorate of 
Combat Developments (DCD) , the Military Personnel Center, or the Soldier Support 
Center on occasion. Responses to graduate and graduate supervisor surveys are 
entered quarterly into a computer via a remote terminal by DOES personnel in the 
systems information division. DOES personnel have programmed the computer to 
compute descriptive statistics and' generate tables of these data for making sim- 
ple comparisons. < These data are used by the training division of the DOES in 
identifying trends in the abilities of graduates to perform or not perform spe- 
cific tasks. This information may be passed along to the DTD, the departments 

he Commanding General of tne Center/School, Data from the logistics question- 
naire arc "reviewed by the Chief of the Logistics Division. The Logistics Chief 
extracts the information pertaining to the I ability ..or uh.i - P^ne to 
' ir^general functional areas (e.g., troubleshooting, use of publications) from the 
logistics questionnaire, summarizes it, and forwards the results to the 
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Mairiteri^nce Department. _ The Maintenance Department , however, indicates that : 
the information provided bri soldier performance is. tod general. To say that 
soldiers cannot trbubleshoot is hot sufficient.. The Maintenance Department 
"needs to know what soldiers, in how marly battalions, feannot perform what par- f 
ticular troubleshooting functions on what equipment arid what are the reasons- 
for the poor performance (e;g., caniiot locate faults, inability to use par- .' 
ticular test equipment) . , 

Another means of providing feedback on unit problems and needs is the - . 

Annual Branch Training Team <BTT) Report. The author of the BTT report, usu- 
ally a branch chief in the training division or the DOES director, selects what 
he/ she feels to be the most important inf ormat ion gathered by the DOES during 
field visits for the year for inclusion in an Annual BTT report . The BTT 
Annual Report is prepared, for the. Commander of the US Army Training and Doc- 
trine Command (TRADOC) in accordance with TRADOC Circular 350-81-1 . DOES 1 s BTTL 
report is not distributed t of other departments/directorates at the Center/ 
School, although the final BTT report produced by TRADOC may be distributed to . V 
the Center /School departments/directorates. 

The DOES collects information from units that cannot be Visited during a 
given year by mailing put* surveys and questionnaires to them. Questionnaires 
and surveys distributed to units by mail include a personnel unit assessment 
quest ionna ire, a logistics unit assessment questionnaire, graduate supervisor's 
surveys and graduate surveys. Completed questionnaires or surveys are returned 
to the DOIJS through the mail, DOES personnel estimated the return rate to vary 
from 16% for the graduate surveys! to ^6% f° r the personnel questionnaires . The 
relatively low return rate for tre graduate surveys might 4 be due in par ^ to its 
length, which varies from -5 to l¥ pages, for different graduates. The personnel V 
quest ionna ire , on the other hand,^Ls only 2 pages in length, and is sent to 
battalion commanders, who may be more .willing than recent graduates to compPle£e 
questionnaires of whatever length. The resulting data are treated essentially 
in, the same way as the analogous data collected during field visits, except that 
the data collected through the mail are not included in trip reports. 

. I 

Although distributing surveys and questionnaires through the mail is effi- 
cient in terms of time, effort, and money, and provides the* DOES with information - :; 
that might not otherwise be obtained, there are serious drawbacks to this method 
of collecting feedback. The low return rate makes it questionable • whether the 
responses are representative of the units surveyed . Another drawback is that 
unit personnel often perceive mailed surveys ^ chores to be tolerated and do 
riot devote as mucH thought to coraplet ing them as they could. 

Another source of feedback from the field on soldier performance is the 
results of Comprehensive Occupational Data Analysis Program (CODAP) surveys. 
The DOES periodically receives print-outs of CODAP results from the Soldier 
Support Center, National Capitol .Region in Alexandria ,'J/A. CODAP surveys 

typically require soldiers to rate a list of tasks on each _of seyeraO.actors. . 

to include task criticality, how frequently the task is performed, and how well 
it is performed. Several DOES personnel mentioned receiving CODAP results but 
only one branch chief indicated that he used the results. That branch chief 
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stated that he extracted data from the CODAP surveys on, the^requericy of per- 
formance of tasks arid -used the data, in conjunction with airier information, to 
determine if the right skills are being emphasized at thefCenter/School - He* 
did not explain just how he combined the CO DAP data with other information, nor 
did he describe how he decides if the right sicilis are receiving emphasis. 

The primary .source of data for matters concerning the National Guard is 
the National Guard Bureau. The Army National Guard. (ANGJ. .advisor frequently 
receives request s_f or information about the National Guard from various -agen- 
cies (e.g. 3 t-he DTD, the DCD) at. the Center/School^ Typically following sufch a 
request, the ANG advisor calls the Nationals-Guard Bureau, which either provides 
the needed data .on the phone or sends the data in written form to the ANG ad- 
visor*. The ANC advisor then "relays Che data to the person or agency request, ing 
it. Among the types of information requested are SQ^results , demographic, data 
on ANG personnel, number of ANG personnel qualified, in a particular MOS, and 
questions regarding the number of a given item of equipment or device available 
in ANG units. 

Feedback At The Center/ School 

■ — : ~' 

Much of the information feedback gathered on soldier performance and train- 
ing materials at the Center/School is collected by evaluation personnel in the . 
DOES training division. .Types of* information . gathered^by these personnel in- 
include performance data on objective tests, perceptions of graduate performance 
student and instructor evaluations of courses taught at the Centetf/School and 
observations of the training process used in conducting the courses. 

4 

Course test results. The training division routinely receives results' of 
tests administered at regular intervals from the test and evaluation branch of 
the Directorate of Plans arid Training (DPT) . 7 These results are received in the 
form of tables displaying the number of soldiers tested and the number receiving 
first-time NO-GO r s on, each task. From the DPT, the DOES also receives completed 
sco re sheets showing each soldier* s performance on each task. _ These scojresheets 
break out the individual soldier/ s performance by sub task and give reasons for 
NO- GO's on each task and subtask . From these scoresheets the DOES compiles the 
percentage of soldiers tested who failed the tasks for eachjeason. The DOES 
also uses the scoresheets to compile the number of tasks .failed by each individ- 
ual soldier. The ^results of DOES f s analysis of the test results are distributed 
to the DTD, DPT, and the various direc to rates/departments responsible for the 
instruct ion . ^ _ r 

When the testing is done correctly in an unbiased manner, the test results- 
provided by the DPT, and DOES 1 s arfaLysis of them, are an excellent source of 
objective feedback to tra in ing developers, especially when coupled with observa- 
tional .data on the training preceding them. On .several occasions DOES. personnel 

• TMfc"iT"re^cW in semiannual or quar- 

terly reports distributed to the DTD and others. But the present branch chief 
has discontinued these reports because of the excessive time req^ired/to produce 
them. V* 
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have observed the testing to be so poorly conducted thaj the outcome could not 
be trusted. in at least one case; DOES personnel administered independent tests 
tb t a sample of soldiers- following a course because of low v conf idence in the 
course test results. The results of this independently administered test were 
made known to the Command ing GenerajL of^the Center/School ^ But wire^not * dis- 
tributed to the department or the DTD. C . > / i 

- Skill Qualification Test. (SQT) results. Another source of objective infor- 
mation received by the DOZS training division is SQT results* 3QT results iri- 
. ^f.f??" 111 °f ^computer print-out are sent to the DOES from the. # Army Training 
Support Center (ATS€) each quarter. The SQT results are broken oat by MOS Irtd 
skill level. The print-out lists SQT items along with the percentage of sol- 
diers receiving NO-GO' s on each item. Itekrs having high failure rates (e~. g.;» 
50% or greater) and the tagks with which they are associated are investigated, 
by the DOES to determine the reasons for the poor performance. The DOES checks 
the wording of the offending item and .looks for. discrepancies betwetNi the task 
as tested on the SQT and as trained in the unit and described in the soldiers'* 
manual . All SQT-related problems identified by the DOES are referred to the 
DTD for its consideration and/or resolution. SQT problems may al^o^be bppagtit 
to the attention of an -individual test specialist working for ATSC bxit co-located 
with the DTD. / ^ 

Interviews with .training division personnel suggest that the feedback pro- 
vided to the DTD and ATSC on SOT problems,, has been relatively infrequent in the • 
past and is becoming less frequent as soldier performance on the SQT improves . & 
What feedback the DOES does prpvide to the DTD or ATSC is communicated by phone 
or in a memorandum. v 

iminars . Training division 'personnel conduct informal 



feedback seminars with officers and NCO's within the first two weeks of the .NCO 
and officer advanced courses at the School „ Officers and NCO's are asked. to , 
estimate their ability or the ability of those that they supervise to perform • 
specific tasks. ^Questions concerning task performance are asked of the officers 
arid NCO's as a gT'oup, and responses are recorded as longhand notes. Information 
obtained through these feed-back seminars is used in conjunction with graduate - 
and graduate supervisor survey responses (collected ^.n;the field) for Identifying, 
tasks that .are performed poorl y by graduates. ' Wien officers indicate that a 
large proportion ot>graduates (precise objective criteria were not stated,) can- 
not perform particular task Satisfactorily , training division personnel 
inspect the training using methods describld in the following paragraph to jdeter- v 
mine if the problem lies in 1 the training program. Problems identified through 
the use of graduate surveys and f e<3dback^seminars are typically referred to 
course managers, of the appropriate departments, the DTD, or to tne Assistant 
Commandant of the Center/School for action or information purposes^ 

■Course evaluations, Hie. DOES routinely evaluates *the courses given at the 

CenterTSchool . Practically all new or pilot courses are •evnliaaaed , with other 
established courses being, covered to a 1 esser degree . Observations are collected 
during" training and" testing to determiner-the ; effectiveness and efficiency o f the 
courses. Until recently;, the observations of training a<id testing were collected 
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■» . - 
routinely using -structured observation forms that allow the observer to check 
for the occurrence of partfcuiar training events arid. record his/her -comments ; 
Using a training evaluation form, the eva.Luator sought to determine it tasks 
Were demonstrate/ and practiced to standard, if the lesson plan was followed, " 
arid if needed training materials arid job aids were used.' Using a testing 
evaluation form, the evalaat^r objsiHrtfe'd resting to determine if the right 
tasks were bSrig tested to the correct standards and checked or . bias by 
examiners' in administering ami scoring , the -T*st . The DOES branch chief respon- 
sible for course evaluation at the Center/School indicated -that these f&nnp are 
no longer used routinely in evaluating training and testing, arid that he j 
favors allowing a subject matter expert to monitor the class and make notes on 
any problems observed. Data collected during course evaluations, are :revie*ea 
periodically, looking for recurring comments Or something that is totally at 
Variance with 1 references (e..g., soldier's manuals, field manuals , ^ technical 
manuals) ," lesson plans, or expected procedure s . When p roblems are iden.tif ieV; 
they are relayed to J:he instructional departments, thfe DTD, or to the Deputy 
Assistant Commandant for Plans >d Operations for corrective action. The fre- 
quency of feedback, of information, on course pcoblems. to the instructional 
/departments varies. For new courses -being piloced, ; feedbaclj may be^given on a 
weekly, basis. Feedback on other courses is less frequent and somewhat Jsr regular. 

Course evaluations are an important part of an overall feedback system. _ 
Tasks -that *re not performed, well by soldiers in the field ^en ate not trained 
well -at the Center/School. DOES Vs must employ reliable methods ■ irf conducting 
course evaluations., Sending a subject matter expert to cla^vith instructions 
to record any problems he/she observes does not consistently produce reliable 
data. This method of 'evaluating training is likely to miss the ""J 6 ?^ 
problems associated with training and ^^ing and focu^on minor echn cal dis 
. cropancias in the course extent. A jvsteUic method for -aluat^ g - 
' has been developed by Al^/aod . is documented in a series-a job aids (Krist iansen 
1981; Kristiansen & Wltm7r., 1982a; Kristiansen S Witmer. 1982b, Witmer, 1981) 

'•Student questionnaires. - InVevaltlat ing School courses, th^e training division 

gathers •Information TrmTsYttdents and instructors. Just prior to graduation, 

Itiiaent questionnaires are. administered in Officer and N'CO courses. Student 

quest ionnaices provide students the opportunity to rate the instructional, ade- g 

potential problem. - / . 

-Th, DOES bases its use of studont^quest tonnaires bn ^jXl^tteftliey 
officers and NCO's are qualified to' judge the adequacy ■ ■ u r a ? " 

receive Is reality, -however, Students -.nuiy not be qual ii iec} to judge auc . 

.. quL" ot the training they rccdLve. It is Unlikely gint sUK - s pos.es . the 
technical sophistication necessary that wouiu a low ^j^f^^^^^i. 
cour-e content or the train ipg and test Irfg. processes iMftnloy^l. .anain . 
nations S^ttd be unduly influenced by the personal it^ or prencntnt ion style 
o fi* ho instructor. ■ J 
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Instructor quest ronnai re-s-. Per iodic al I v , the DOES ut.il tzes instructor 
questionnaires to get the instructor's p.-ppectivoon the adequacy of the 
course design and the quality of the training materials provided. Instructo-r 
questionnaires are most frequently used in conjunction with piloting a hew 
course or during a speriai study" of a particular course: they are hot used ; ^ 
routinely in evaluating training. Instructor questionnaires are either nailed 
to instructors or handed directly to the instructor following a class. In 
either case, the instructor is asked to complete the questionnaire independ- 
ently and return it direct! v to the DOES ._ L'nt'il recently, the DOES tabulated 
the instructors 1 responses on each item for inclusion in quarterly or semi- 
annual course reports. We were not able to determine how the results from ■ 
this questionnaire are being used at* the present time. 

Training inspect ions . Co-located with hut not directly affiliated with 
the DOES is the Commanding General's training inspection team.. The primary 
function of the team is to periodically inspect the training of every unit at 
the Center/School. This includes activities responsible for institutional 
training as well as active operational units. In the past year, the training 
inspection team has inspected over 600 blocks of instruction. 

A standard training inspection report data collect ion form (Appendix B) 
is used in inspecting the training and reporting the results of the inspection. 
The" data collection form includes items to be observed during, the training and 
testing. The form has a place for recording whether or not the event described 

in the item occurred and provides some space for inspector's remarks and 

recommendations. The space provided however is inadequate and does not encourage 
the training inspector to make comments;. A special inspection form is used for 
inspecting physical readiness training. Al\ other training is inspected using 
the standard inspection report form. 

The training inspector observes t he. tratntng and testing as 'Jfiry occur, < 
recording answers to specific questions relating to the quality of the training. 
When an answer to one of these questions indicates a_ problem wit^ training or; 
/testing, the inspector writes a short comment beneath the item, explaining why 
the question was answered as it was. Additional comments and recommendations 
are recorded on the last page of the inspection report. to a : last step in 
inspecting the training; the inspector assesses whether or not the training 
achieved its training objective arid record:; his/her assessment i n the report. 
This assessment is to a certain extent subjective, and taken alone may not _ 
adequately, reflect the effectiveness of the training. Following the in spec ion, 
the inspector briefs the instructor on the good and bad joints of the training. 

I-ach completed inspection report is reviewed by the chief of the Training 
inspection Team for completeness and internal consistency. The Wa ning inspect 
tor: U>?H the.rlass In on a large chart a s ha v f ng been in spec t ed , wh eh 1 i st .. ^ 
the unit and class inspected, the instructor, t he I n^-f t n r n o ve ra 1 a sse 5«*nt 
and the date of the i tispee t ion . I1w completed report t »u n MUd 
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Copies of the in. .-per : ion rer-ort .if. out to ■Omrii^ding Cone ral j of _ t he. 

Center/ School .in.! to the -nil re spun : 5 ■ e fr the r r a in::t^ ti-it^va^ . irrspe/te .i ; 
Depending on the nature of the pro-l.r.: idetu : ! ied:- copfcfs of t.he/r«epor t r.iy ' 
also So sent to the DPT, the 'CI', or t he 3 <••»;.->. 1: This; iteration; .say he use J 
hv the units to sivJifv their train; v\ .! '-.»the :?:pthe'.rrrf> or other Syhool 
u'irectorates.'departnents to change fa. torn advers./ly impacting on the training.. 
Some units al rso utilise the information provided V. t.ho repojis In keeping track 
of the .state of training in their or.it. 

; /.-:.- . . 

» p.roferisionaiisn jurve% : s. i Tlio personnel division administers professional -ism 
survevs w "oTrtcerT'anJ NCO'« at teiid ihg • t he oWi-cu* and. en 1 i-sted advanced courses 
at the School. The survevs. are desired to ^tl»er .ti'.ie opinfons of officers and 
SCO's in several different areas, i nc 1 inii n .: '. t he p ro :es s iona 1 quali f ica t ion:, o f ■ 
other officers, carver perceptions, career intentions.; career sat isfac.t ion,- , 
training considerations, and the perceived q«attty. o,f naw recruits. . Demo graph ic\ 
data are also collected for each so Id i e K respond in « t.o f he survey;. The responses, 
to these surveys are anaivied it os-bv- 1 1 err in or dor to i den t i,f y _ tr er.ds, i,n the 
data sUggestive of personnel pro hi ens: IV- so r i p t ive "s t a t i st lea . r each purvey, 
item are computed periodical iv and a roil-up of the s'umru r ized r«sul : t.s is;, pro- . 
duced in tabular fohn. tfio descriptive statist ios are used by i*C personnel 
division to Identify problem a reas . ProKlemf; thus identified referred to .. 

the directorate/department who is in the Best /plis.l t fe?n ;to do something about tacn 
(e:g.'. Directorate of Combat tv ve iopment s i. So 1 d ie r ■ ?>u rP n r t: Gen t e-r , ttll ?c: rpen). 
It tin- directorate/department is unuillliU- or unable to resolve the- problem-, the 
•ieputv (JomSandinv. General mav be advised. • Problems at feet ing the en t ire command 
are brought to the attention of the Cent er/.chool lenS.d trig General . If the 
Commanding General agrees that 'the problem -hould he- ..idd re sscd -, ■ t hen ,ht> d t r ec t s 
the appropriate d i ree ro rat e;/ de pa r t men t to fuse ..et.ion to solve the problem. < 

TJ.it. i lUMluet-iiMi- and. ^al^yis . _ 

The larpc uiinmt Of data collated hv the IK** nec 5 s , i t a t c s. t lia t the' data. 
!». reduced prior to distribution to other agencies.; The primary Strategy used 
hv the BOKS in reusing, the .lata eonsists of .summur.i ,.in K .the ^lata ove r ?«V ■ 
.ore variables anil tabulating or c ros,, tabulating the result^ ; The tabu 1. r I. A 
r.--„lf ire tvpic ;il ly 'expressed as a ! re-.uere v count or as a - pe r con t. a «e « f t lie 
,t re' j on Kiit* example ; course te.t result, mav be expressed in terms oi 

1 • ,e eeit.ige of soldiers tested who pas -.-d -t he test 1 o r eac h t a , , . an d t he 
••i ere'h ape *hU failed for each of -several rea^n, (e , ; . 1 au It v P roe edure t-^ 
„ iJ-'): "The tabulated result,' At? either forwarded jo the appropriate 



,,,,„,,, ,!iowine. the agency t o_ d raw t he i r owi, eon. l.e.lon, , or el t he j** 

' hv 1 lie i e-.ul i ■; , an.! .ommun ieat es t lie It int. rpr. 

'*. ii •-.«••; a pre- 

I lull ill 1:11' iiriMMnii:. i" p."- j- - - - ■ - tj . . , ■ ■ m 

iili^il eciterlun as a w:.lll s _ n i ,1 d s ,u i i v i n , s J *, 1 hie att I r oh (, . . . . i, 



i.iHiti.ies prohieiHs Su, K< -,t ed ;h v the ..'.•... !.' !/ . . *i V)l ' i e . ,r,- 

|:(I| „ M of the prohl.-:ns to .he v. I" so-,.- , a •.- :, , • t he \*M. . u , . a pr. 



f :n: or more oi the re,p,,H,esao an item su,,e..i ih.it a proh .i h th 

ii.ii:-: iaeht.iies.iHe proiUe,,, i , he .■,„,,,,;, i, n end 

inio ,J.:.U iudie (!t ing a prohiemi.ui, I- U,^,, ^^ : •? ' • 

: _:: :. ; ?; i . : . : . • . . iiriii; t j.ih ! .if ! hi I hr r Hon it it -iff « • i -»i 



ol Ju>v units served hv i4fi. hent-./:. 1 • appea, ia.be in-iea-.tn^ in 
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severity ;;vir i.~* . . i/ri:;:-;i'-.' ' re rid:*'; . hear I y otfe-half of -the division arid' 
" r.mc!. •-'•i : H 1 e rv : v i.:e *}•>*.;': ci t ha ? ' t he ;< lovked at trends . in the ddjid _as' a . 

nuKis o f i vii ; :: t if v pi*o!w.«.' >- -;.' ■ .->.N*-o of the chiefs wore afcl<r to specify pr^e- 
..^iselv hov t;a : y dot i. : r - ine t.'.o " ^vVi.^'lCiiV^'-.of a .ti^nj. ' (?Re of tfefec irit er^viewed , • *• 
aJ-Jtti d t:.at\'at 'east • !' S r h ; ^i!ic;'4 i ? vi nV* trends wa $ done ; fluUj ec : t i v|lv bas'od '•. 
o:i his host jwd^i-at. ^:'i;yD. r : v^;vs' ^llii 1 *. in "t!;^ 'cbu^^ of the .^nterytevs sug- ■ 
•V-sts that the. APE 5 does.^: h A Si'3 eVea'r- ~\ii -Jektruv-. : .ttf/s t •.the*': cons is t eric ly- .use 
in i ti on t i f y i .: x . p r a V 1 o nr; . " 0 ro r. i' jc i :~p i e , • ' «j o v e ra f i ft Co r v is? we e s expressed di f f i\ 
Cnlty -in dcter^n in:: ho 'uv ro^u^: fcv ' a; ppo-i.-lt^^tad .C'c pentioned before it 
colli, i' ho c-^iil^i'd svru'cf eTieor>>s to warrant :f er € birr act ion, 

■ © • ' ■ ■ . _ ■ . " _ _ • • _• • 

£V,o rO::S •appears r^'use.::St at. i ^ticjii prncedu/rc* .for data reduction on • 

vr r\ ' 1 i^i tod ' ba° i •> . The' iis'i.-. ' o"? tat 1 • .irtal ii* for arriving at conclusions; 

and :?..ik co-pa r i s.mi;, so i ir - 1 1 ■ he 7 jthiOi,' * t>< -lori rath or than . the 'rule . During • 

t he ; iriterv lews only oho x i n :;'fja-tt.ft- _ wai; *^l ; r,r ^.ni^ irr which statistical analysis -. 

was ii-.i-J bvJ^ES personnel; Co dr-iw ^ii;.i^v ; i' regard in £ the 'data. Although ,v 

IXv.y, pvrsofiiiel; ; ?cf}ue;;t l.? c o^V^ rL ' rifs.ultfi a^.ro Ss uit : t k st*a 1 1 st ical procedares ; 

An- seldom, u^od in rr-ikin'^ t ho c -:-;m r i -vwa s T ^ It appears t.'hat comparisons a ^:'- 

t-vvvt often ^adi- hv cyjifrt nih ' :h:e t da: :t' ■ in\t;:e ".1 fey.hg -of on^'s experience and . • 

di-:- id ^subjrvrt ivel y) wither" the d i ffer-o.no^ of discrepancies are ; of *suf f i- 

rient r.a.^n it tide ' to eon ; ;tttut«- a problorn^f t? ; . is r%jh [rrtivo a^sessflcnt .'of .results.,* 

w:;l,'h varic; fron one person to the next, is a poor 'sabs't itut>e .for s tat 1st ic^l 

atial vs i's oi. otiier ohjoet i ve Ue'e i s lo ; 'n-n..ik inV, procedure^ f ■■ aad'- p goc^raUy nbt' 1 . 

.ieeept.oi! l ov sv/itffe analvsiiJ or her *ricrr.L L-'.r o f the. ana 1 v t i c . cofnrhun i ty 

V -.ri ulne: : . Ahd Pvl iah i ! : tv n ♦ ' Te».:d!>ae k * • "• . . .." , 

- ■ ■ * 4 ■ • . ■ ( > . ■ 

Iror. tlio previous d i -;e us s \\m o : t he- tfodb^ick r:rt !;.o d-; or,ployed by .the ; DOES", : 
it i\ clrar that t he usefulness and roi iahfl itv'of thi- fiiedh.Kk provided 'to tte 
Cmlri / .VcUooT is not al 1 hat 'it eould* >;-f . Hst irsato ; graduate pL-riornvince by 
^.radiiat i*-s ana* th.-fr :aipe rvi -;o r s Si v hot accurate. Vui* li thv sane applies to . 
st.id.-nt. rpif-rt ionnai res-; . Tf^ ^iu«' ;; ;t ionsSa sfceJ on r; u<« s ,t i onna i r o s . and during inter- 
vi.'V. a r«- olteii iiiiieif .frijo v;eiv.-ral with 1 i t t 1 r ebh s 1 de ra t i On. <>4" the actual f.eed- 
j.aeV iicei!-, oi d i r« : e r orat es/depa r t hii-ri t s at the C>n t e r /Sc K)ol . Interviews con- _\ 
duct .Vi Jiv tNi fa^V: the st ructnro t hat M nee e s>a ry loV a pr iinary,- feedback 

., lMiro < a N4 l <!,, u«>t adoq\iato!y «vvi'r battal ion personnel at U-vels lower than the 
,,,,,,,,1 lev.-!. (.nirsv' evaluations iti'whirh « >ie.e r va I [ iih r; arc not structured, ancf - 
which depend mi a s-ihje.;!. rttit tor o.y.jiori to r-^r.-lv recotd^what iu-/she nee>:are-uot 
MM-.iv to -Mnvide iul foedbac V to trainin>-. dVvelopers'.- ^oibject nvitti/r i'X- 

peits'of ret* iMM-nne ,s.> involved in 't he ' sub i,i-<- 1 r-utter HolMyi t.m^t t ha V they' all 
hut icnote tito t:v»niior: P itf which t,h>- training I s • ,;on d ur t M . Sue h probl«*mJ» tencl 
to rod.no thr use'ititho;.-;- u ! the. 1 -i I i a O dliac i to t ye ( -n t o r / lie* 1" lOO 1 s . .Hie/ _ 

u/oiuini'S'.'.ii t he' IredhaVk i s ; f iirt »ii-r d i wttfl sih-d h\. 1^;:;;', laek.ot .« t andardi zeJ 
ohj.-et ivi : proredure-;\£or aiirt 1 y?. i m\ "t ho il data; ident living trend:;, and drawing 
,„.„■ lu-. ions «»i ::u'-!n^.UvI' ion-; on t tio basis •>! fh.-ir data. 

'•' iv.'.j.iti. ;tiii. ht.»»,i < ,l.,...Wni iiuii-f ^iliiiviv Si-.i i.i! >Ku 1 rh.^u!,,!ivlsigH 
iirti...: .■<,,,,,;.;.•,! i ,!.-,.• .• In-th,- .Ufaj ^tij-y V:M I r;1i-« : >'»„•„ 1 ....v^. a.b<n. t 

nr< ; ; ( .,H,„ili : .v „f ilu- ii,j;irm.,Oon j^llsU ( |ai t «.;f_; \».r.m.;h «V» <l/.!4 vl r, loji 
■;'iOi.j.-."in.ii..ii.-.i Hut'iho I.O i.-v.l .^H .>1 tl* i. r «l!..ir|. m („■ n-lUMf. 

' f - . ** • v 
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Course test results arid SQT results were perceived as being reliable as were 
questionnaire and interview data collected during field visits* Only two. 
sources of data were questioned with regard to their reliability or usefulness. 
One branch chief questioned the rel lability -of . the graduate supervisor's sur- 
veys on the basis that the results maybe affected by how well the supervisors 
expect the graduates to do. toother branch chief expressed confidence in the 
reliability of the graduate Supervisor's surveys but lacked Confidence in the 
ability of graduates to estimate their own performance on specific tasks as is 
required by the graduate surveys. Professionalism surveys were considered to 
be of limited value as a means for identifying problems. 

Feedback Needs 0 . 

■ 1 .• / • • . 

Feedback from users . >tost of the information collected by the DOES is 
used for identifying problems associated with processes and products of the 
"Center/School. As mentioned earlier, the DOES typically refers these problems 
to the directorate/department who is in the best posjtfion to solve them. Prob- 
lems are usually described in narrative or statement form in a report or mem- 
orandum that also contains information briefly explaining the source of the 
data. Normally the DOES does not make recommendations on what actions should 
be taken to address the problems. The solution to the problem is left up to 
the responsible directorate/department, as is the decision of what^action^ to 
take if any. In one sense, the DOES can be said to turn over whatever prob- ■ 
lems'it finds to the responsible directorate/department.- The directorate/ , 
department, in turn, may or may not address the problem and is not obligated 
to take any action whatsoever to eliminate the problem. 

Generlllv^irectorates/departments- receiving 'information about problems 
from rhe DOESdo not inform the DOES about what steps, if any, ^ey are taking 
to eliminate the problem identified. Thus the DOES^receives^ery little feed 
back from users on how the Inflation that it Provxdes is being js^, or «ven 
if the information is being used. .Occasionally a use? send.a_ reg^ 

the DOES indicating their agreement or disagreement with the hr^nrh cMef ' 

Ind describing what they are doing .to' address the problems. One branch chiet 
indicated tSt whenever he reported problems to a directorate/department he ■ 
' reaiefted ttat tL directorate/department, by a specified date inform him of 

were found during an Inspection of a particular D ^ then the DOES can 

cases the DOES may inform the Commanding Genqral s ottice cnai. . , 

action has been taken to -eliminate the problems. . •# 

The lack of direct feedback , from directorates/departments regarding hpw 
they^Corlf they are able .td ft) .the -f^J**?*&& ' S 
a major impediment to obtaining better and more useful feedback ls ^ tire l y 

regular user feedback, the DOES may assume the feedback tney pr 
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satisfactory and-is useful in correcting:. Center/School -problems. Statements by 
DOES. personnel expressing confidence in the reliability of the feedback they 

provide, and their reticence ^.n identifying a d d i t o nal_fe e d^jc^ni^a-^Lnd ica-t-e 

- - their- overall— sat is faxrHxm -w±th"rhe— f eedbac k THey~pFo vi<? e , User feedback is. 

needed to inform the DOES of which types of feedback are most useful and to 
identify additional feedback needs. 

Additiona l feedback needs.) DOES division and branch chiefs indicated few 

areas in which they needed additional feedback. Two of those interviewed in- 
dicated that they already were getting more feedback than they could handle. 
A third expressed the opinion that "present levels of data collection activi- 
■* - ties were-- placing excessive demands on units in the field. 

' Of those exp re ssing the need for addtriorrri-f ^^h^rVj Fhi ^ra~»a rrrprii wst - p 

task-specific feedback regarding hands-on performance in the field. One sug- 
gested that this feedback should come from hands-on tests administered to sol- 
diers in the field, but another felt that DOES' s did not have time. during field 
visits to conduct hands-on tests. One of those wanting more task-specific 
feedback was especially interested in getting more and better feedback on gun- 
nery performance. He perceived the need to identify individuals who contin- 
ually fail to qualify on various gunnery tables a^pL exercises and what types\ 
of engagements caused soldiers the most difficulty He suggest^ -that/ each 
time gunnery is conducted the following kinds of information should be recorded 

and made available for analysis : _ (1) type o f „ g unn er.yL,exprc ise^or- -table-^-- — 

social security number of the soldier(s) participating; (3) when and in what 
location the gunnery occurred; and (4) what engagements and parts of engagements 
were failed by each soldier.- 'Three interviewees indicated that they need more 
informat ion on the availability and frequency of use of training materials , 
including technical lessons, devices, and publications. ^Another need alluded to 
was the difficulty in" blijta in^ particularly in 

-\- the reserve component. , , . \ 

Summary _ w. • /V '.";,'*'" 

— DOES' s provide much of the feedback available to Centers/Schools . Various 
■* methods are employed by DOES'.s to provide this feedback and* distribute it to 

tjhe appropriate directorate/department. Some of these methods, however, produce 
feedback that is neither reliable nor useful to the directorate/department re- 
ceiving it. Aside from the methods themselves, inadequate analysis by the DOES_ 
and lack of direct feedback from users are .the factors that are most detrimental, 
f to* the reliability and. usefulness of 'feedback. . " ' . 

The types of feedback discusWd in this section are listed in Table 1, 
. along with the types of feedback available to the DTD, which are discussed in 
the next section. Also included in this table are the sources of feedback and 
thSi initial recipient of 6ach type (DOES, DTD; or both) . The table is intended 
to summarize both this and the next section and to bridge the gap between them. 
While reading the next section, it may be useful to refer back to Table 1 t© 
obtain a summary of the types of feedback already discussed. 
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Table 1 

Summary of Feedback Available to a Typical DOES and DTD 



ypc-of-FendbacIc" " "~ "~~ ' " Source v Recipient 
xternal : 

: ■ ■ ■ " - ---- % I ■ 

Graduate .supervisor surveys • > Supervision of recent graduates * DOES 

Graduate surveys ^ " Recent graduates DOES 

.Personnel questionnaire . 'Battalion commanders DOES 

Logistics que$tioiinaire : Battalion log off, raaint off, or XO DOES . / 

ARTEP questionnaire Battalion operations officer- v DOES 

Informal interviews \ Battalion and company staffs " ) _COES 

Trip! reports and ^extracts Center/School personnel visiting field units , DOES & DTD 

Unit training materials Field units DOES 

CODAP survey results Unit personnel through Soldier Spt Ctr ^ DOES & DTD 

SQT results _ # Unit personnel through Army Tng Spt Ctr DOES & DTD 

Informal, unsolicited feedback Unit personnel DTD 

» v 
iternal: / 

. • ... . ... * ' : H- 

Course test results . Dir of Plans and Training DOES 

Course evaluations . k Observations of classes * DOES 

Student questionnaires Students DOES 

Instructor questionnaires Instructors DOES 

Informal feedback seminars Students (officers and NCO's) DOES 

Training inspections Observations of institutional and unit training DOES 

Professionalism surveys , Students (officers and SCO's) DOES 

Program of Instruct ion* (POI) ' 

changes •_. Instructional departments ^ DTD . 'r 

Pilot teachings arid student trials ' Observations of initial course administrations DTD ;•: .■.;> 
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■ ■ ; SECTION IVo ::: ; : : ; ; : ^ ■ - : : ' 

-DTD FEEDBACK SOURCES MD NEEDS ./ " . ; ' 

In this section the feedback needs of training developers are addressed 
through review of the results of interviews with the DTD division and branch 
chiefs. Of the six division chiefs interviewed, four indicated that person^ 
nel in their division have a continuing need for feedback ori the performance 
of soldiers and^use of training materials. Two divisions are primarily' in- 
volved in the. production and distribution of training materials rather than 

in- their* "deve iopmenx-. Personnel ±h jrtrereTai^tston^^ 

direct heed for feedback, and thus they were not interviewed extensively. 
/The information presented here is derived primarily from interviews with 26 
personnel involved in task analysis, the design and development of individual 
and collective training programs and materials, and the development and manage- 
ment of training devices. As argued earlier, these personnel are representa- 
tive of those in any Center/School DTD. ^ 

In order to avoid attribution of interview responses to specific individ- 
uals, the information gathered is not organized by division or branch in this 
srectiarr. Rather, the results "are~or'gani:^ed — f ±rsrr~by types of feedback currently 
available and secondly by major feedback issues or needs that arose during the 
interviews. In the descriptions of feedback currently available, emphasis is 
...placed on types^bf feedback from spurces_^xterhal_.to the Center/ School (i.e.^ 
field USiits) , but internal feedback is also briefly addressed. No attempt is 
made to describe these types of feedback In great depth or to detail the ways 
in which they are transmitted through the DTD. Instead, brief descriptions of 
them are given to serve as a basis for the follow-on discussion of feedback 
needs; These needs are described in terms of major problems or issues which 
surfaced from the interviews, and types of feedback are further discfisfeed and 
evaluated as they relate to these issues. Recommendat ions for ways to satisfy 
the feedback needs identified are developed in a later section of this report. 

Feedback Currently Available - • 

Trip reports . A common way in which DTD personnel obtain external feed- 
bacK is through reviewing written reports of information gathered during visits 
to field units. These trips are usually taken and. reports written by DOES per- 
sonnel, but reports are occasionally received by the" DTD for trips taken by the 
Commanding General or other Center/School personnel. All DTD personnel inter- 
"viewed who have a need for feedback from the field indicated that they receive 
copies of DOES trip reports or extracts from them, as described in the previous 
section. While most interviewees stated that useful indications of problems in 



,thei_f leJLd__ar_e sometimes 4frive j_^-E om ^rip reports, several criticisms of the 
- utility of such reports as feedback were consistently brought out. 



The most common critique of trip reports mentioned- by nearly all Inter- 
viewees is that these reports do not usually provide enough specific detail 

about problems existing in the field and recommended solutions to them, ftajor 
problems with the quality of soldiers or training materials are briefly de- ' 
scribed in trip reports, based upon observation of training, administration of 
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s^r^eys or iritferviewSi and informal interactions with field unit personnel . 
" :iui. ^BpijS^;y&te often not described in suf f ipient detail to allow training 
^.>dfe1^-0^i^i.'^.^^rmi^„what to do about 'them. An example of this situation 
• ' 'provided by two interviewees is the statement from a trip report that "lieuten- 
■ . lants^ can ' t read maps . " Training .developers need further information in order 
' 'to address this. problem, such as the time frame during which these officers 

completed institutional training and the subtasks which cause the greatest 
. '• difficulty. With- such ihfoi^rtion available, training could be redesigned 
'• to address 'fifie problem. / ' S 

Several other •probi^ms'-wit'h the' obtaining of feedback from trip reports 
•were mentionedjby more than one interviewee...:. Five interviewees expressed 

; " concerns aboutf*the validity of information p&vi'ded in trip reports. Trip 
reports of. ten summarise general observations or opinions which may. not be \ 
completely accurate. Trip' reports also often do not delineate the extent of 

.' a problem identified, that is, it is. sometimes- not clear whether a problem 
is unique to a particular unit or widespread throughout field units. It is 
als'o sometimes unclear whether .a perceived problem is" based on the opinion • 
of a few individuals or represents the concensus of numerous .field personnel- 
Two Interviewees brought Up another .concern with the validity of DOES trip - - 
reports. This is that information gathered in the fieltf by personnel per- - . 
ceived to be evaluators (i.e., DOES personnel) -may noV" accurately re-fleet the 
state 'of field units. At least some units tend to' p/t their best foot forward 
for evaluators and to let these personnel see what they want them. to see. 

Another problem with extracting feedback from trip -.reports £s related to 
their distribution. While all DTD interviewees indicated that they had seen 
. DOES trip -reports, the reported/if regency of such reports .varied . Three per 
sonnel reported that theV had not seen a DOES trip report In oyer -a year, and 
several others indicatedVhat /hey saw them irregularly and had not seen one 
for several months. Seveifl interviewees also expressed a suspicion that 
trip reports prepared by personnel in agencies other than' the. DOES are not 
distributed to all personnel having- an interest in them. delated to concerns 
about the distribution oftrip Sports are conc^., about the^ timing . 
Reliance upon trip reports for feedback sdmetimefV?es not provide information 
wtthin the time frame that it is needed; For example, most toterviewees indi- 
cated that they have an opportunity to provide questions to DOJS ^|fj|™f to 
be addressed during upcoming field visits. But one interviewee indicated that 
, responses to such questions are not included in trip reports until four tp 

eight months later' and two- indicated. that responses have never been • received . 
Anotner problem here is that DTD personnel may not know what questions -to ask 
if they do not know what the major concerns ±U,the field currently are That 
^ trip reports do not provide continuous., feedback in which major problems are 
: identified Suring initial trips and followed^' 'with more in^S^qaestioning 

integration 



of data 

. The discussion above shows that, while ■ review of trip ' 
and sometimes useful method. of obtaining external feedb ^f'f^ff Quired 
have se'veral concerns about the quality, and usefulness of- the data acquired. 
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The. review ;of trip reports is in general a rather onsystemiatic::way::of gathering 
feedback. Problems with trip reports and ways of remedying them are farther 
discussed in the context of general issues in the rujtxt section of this report.: 

Field visits . Ah* alternative to training developers 1 relying on reports 
of trips by other personnel to obtain feedback would be for vthem to visit field 
units themselves. Interview results indicate that such visits occur only 
occasionally. Less than half the DTD personnel interviewed indicated that they- 
had visited field' units, and those who had reported that such visits were. rare; 
The number of visits per branch varied from one or two a year to one every • 
three or four years. The DTD does not have the resources for extensive field 
visits, since that mission has been primarily given to the DOES* ; The centrali- 
zation of field visits within the DOES is* an efficient use of travel resources 
and minimizes the disruption of field units by visits from Center/School per-^w 
soririel . However , restrictions on visits by training developers to the field 
does create some problems, many of which were 'described. iri the earlier discus- - 
siori of trip reports. , 

~ 7 £el iance^bn" Te1e~<lb~ack gat hered^duFihg Tfei"d" visi'ts" by ^o^Ber 'per ^ 
suits in training developers working ,with secondhand information. As described 
earlier, several interviewees indicated that this lessens the utility of feed- 
back, "since the information gathered is of questionable validity or riot iri 

sufficient detail. A few interviewees indicated that they will not accent any. 
feedback as accurate unless they gather it themselves directly from the source 
in the field. Whether this perception is correct or not, it leads to an atti- • 
tude which does not foster cooperation between DTD T s and DOES Vs. Nearly half 
of the DTD personnel interviewed indicated that there is a need for increased 
personal contact between training developers and users in the field. Sugges- 
tions for' increased coordination of field visits between training developers 
and evaluators are- offered in a . later section of this report. 

. The" few trips that interviewees had made to the field were perceived as 
very useful. Most of these trips had been taken by front-end analysts during 
the early stages . of training- development. Extensive surveys and interviews of 
field personnel were conducted during many of these visits, although, this fact 
has often been downplayed in Order to avoid time-consuming coordination b£ : ^ 
survey forms T Task analysts have obtained updated validation of task documen- 
tation (tasks, conditions, and standards) during, these 1 :. visits^ arid iri most 
cases tSey feel that such information- Is more valid than that gathered through 
ot^er sources discussed below. '. r »_ 

If more personal contact is needed between training developers an4-users-z 

iri the. field, an obvious solut iori^wou-id be for DTD personnel to more frequently 
visit the field i Seven of the 20 interviewees directly stated that they need , 
to visit the field more frequently, and, several p titers indirectly indicated 
~ such a need. Marry- pofcent-ial-gdvant'ages^of - sucri-visits were ment ioned' -Imple- 
mentation assistance provided by training' developers might increase the use of 
training materials* Observation of task per formarice in the field would lead 
to more objective validation of task documentation than that obtained through 
surveys and interviews. ; Training developers would be perceived, as. helpers and 
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would obtain tore useful^&forraat ion from field personnel than training evalu- . 
a tors do. But despite all these potential benefits , not all interviewees 
favored increasing the number of field visits.. Many • branches are understaffed J 
and do not have thty personnel resources to conduct such visits. The reliability 
of observations made during field visits was also, questioned. As mentioned ear- 
lier, many personnel feel that units let you see only what they want you to see 
during visits to the field. Field visits must be properly "resourced and de- 
signed before their number is increased . 

■ in summary, interview, results indicate that training developers rarely make 
field visits. But when such visits are made, the feedback derived from them is 
generally perceived as being very useful. There was a wi despread but not uni- 
versal feeling among interviewees that their opportunities to visit the field 
should be- increased/ Methods are needed to increase the interaction between 
training developers and users in the field within existing resources. Possible 
approaches to this are discussed in a later section of this report. 

CODAP results. Another major type of e^Sal^dbac* '^^^S^^LZ'. 
• used by training de velop erl "is the^ results ^rXomprehensiye OccupartOttat- Pata 
MalysL Program (CODAP) surveys. As described earlier, these surveys are con- 
noted periodically by the Soldier Support Center, National Capitol Pegion £ : 

espondL e trrr;teTacnrsron g Tvariety of factors, such as how critical it 
H hn„ freauently it is performed, and how. well it is performed. The results 
of'such surveys are of primary interest to task analysts, and these personnel 
-diretly^ 

^KnliJes'of^ 
of the data from Soldier Support Center as needed. These analyses P > 
descriptive breakdown, of the data accord ng^o s if led criteria. ^ | 
a listing can be provided for all tasks meeting V task analysts 

frequency and criticality for a given MOS. The result s are £ stituti0 n and 

and by task selection boards to select "f%^,"f^f used to deter- 

in field units. CODAP results thus provide feedback that can be us a 
mine what tasks should be trained where. . ■? 

I The limm de.eloper/liter.l^^ti^.d «v.r*| concerns »« jj g»J|j 

vey addressing officers tastes wiuixii ? ^ enlisted tasks are accom- 

^niy once approXi*^ two ^^^.^^^^^^i,^^.^^ 
plished only once every ~three |eaxs^ Due to. c p , ^ <T he per- 

today's Army, more continuous feedback on task pert lncr eased in order to 

sonnel resources of Soldier Support Center wog|Jav| " with their 

provide such feedback. Another_ concern with -"ff^f^^ctive estimates 
' perceived validity. . Some interviewees perc.ei ved t tat^subjectl ^ of 

gathered in lengthy surveys may not be valid ^icators ot Xeivatic.i of 

-sks or: * proficiency with which they are measure the fac- 

tot !ddreSS C Llb^ luSeyT ^'further support of this position, see- 
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Burnside (1982)1 A final major concern with CODAP data is that they are hot 
'always available* ' No CO DAP results are. available for Reserve units* arid they 
are badly needed,. R esu lt? are also hd t a ya il ab,le f b r s ome low^-density MPS ^_gj»___ 
and for tasks performed on equipment which ha~s recently been introduced to the 
field, Sinde 18 months aire required to receive data after a task list has been 
submitted'" for a CODAP survey, needed data are sometimes not a*\raiia£le. Or they, 
are not available when needed, as discussed under the timeliness issue above; 
The stated- concerns about CODAP data can be r€J l? t€J£ ? to lack of personnel 

resources to perform surveys more frequently and to check their validity, and 
to the lack of sufficient computer priority to perform analyses quickly. CODAP 
surveys provide an economical but not completely satisfactory way of obtaining 
task-specific feedback. 

Because of their concerns with the timeliness and validity , of _ 03 DAP. .re- 

"suits, training developers sometimes attempt to supplement these data. One way 
in which this is done is- -through field -visits, as discussed earlier- Visits of 
DTD personnel to field units are often called job site visits or interviews, 
but they actually often involve administration of CODAP- type surveys . Su^h sur- 
veys address a smaller list of tasks than is usually addressed in a CODAP survey" 

and they sxe. frequently adSta is tered 

deveioper is available to answer questions . Use of these procedures leads • 
-i¥afefe^-deve4ep-& rs to perceivo tl ta^^ehe^^su lts of t hese in t e r vi ews arc more- 
valid than results of CODAP surveys. Interviewees indicated that if a discrep- 
ancy existed between CODAP results and results of a branch-conducted job site 
visit , the latter would be accepted as more accurate. Until the CODAP survey 

system can be made more flexible and timely, training developers will rely on 

data gathered during field visits whenever possible^ At present there is .no way 
of integrating these two sources of data and checking their relative •val idity. 



SQT item analysis . Field performance data are rarely available as feedback 
to training developers. Most of the currently available feedback identified by 
training developers interviewed consists of general observations, informal com- 
ments, and subjective survey responses. The only relatively objective perform- 
ance data which have been available as external feedback to^the DTD are Skill 
Qualification Te§t (SQT) results. These hands-on performance test results are 
provided to SQT develop ers in the DTD (and to training e valuators in the DOES, 
as discussed previously) by TRADOC* s Army # Train in g^Siip port Center in theform 
of quarterly and annual computer print-outs, or item analyses. These analyses 
provide feedback on the percentages of soldiers passing and failing each task 

tested tfn the hands-on and written components of the SQT. This information is - 

used to revise the test for future administrations. Tasks which are consistently 
passed 'by' 100% of soldiers tested may be eliminated on future tests. Tasks which 
are failed by a large number' of soldiers may also be eliminated, or the question 
nay be revised if its form appears to have created the difficulty encountered. 
The item analyses provide important feedback to SQT developers as to the adequacy 
of their test construction. They also provide performance data which could be of 
use to other training developers. Two interviewees not involved in SQT develop- 
ment indicated that they need increased access to SQT results in order to identi- 
fy potential training problems in : the field. o 
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: However, this need for increased : access to. item analyses throughout DTD is 
apparently a moot point. Current plans call for formal administration of the 
hands-on component of the SQT anci reporting of results to be eliminated by 
FY 83. Item analyses of hands-on performance test results will thus ho longer 
be available. Results of written knowledge tests will continue to be available, , 
but these results likely* will not _as directly indicate where training problems 
occur in the field. The amount of field performance da tav available g$ ^.feed- 
back to training developers will soon idecline from little to none. 

Informal feedback . Several of the types o v f feedback currently available 
to the DTD Can be grouped together because they. are not regularly collected ; 
through formal means. Included here 'are responses to questionnaires included 
in the back of training materials, requests for changes in training materials 

..on---DA--Foms---2Q2a\ r --Ainso f-rom-f ieid- personnel-,- - 

meetings at conferences, and information gathered due to turnover of personnel 
in the DTD. A common characteristic shared jgjjn^ 

"is that ;"tHey"o53^nL^x:equefftry prbyi'd'S'"ui"fef ul~ inf orm^tion to training develop- 
ers. Almost all training manuals include short qge^rionna ires for usfirs to _ t 

___._.._^-_ - . - ~ ■ ' - V ■ — 1 — — - t^I' . 1 t J — . J 1 ~ * *• ^ A #-V*^#- r*iiV>l. 



"conipIiiEI "and return to IrTaitUny^eye^ indicated that such 

* questionnaires- are" i_rel*dt5m -r ercrrnedT mo S"tr e srimates v f "re turrr rare r were ' -rtn ■ the "~ 
range of one or two per year. Requests for changes in training publications; 
are, received slightly nio re frequently; interviewees estimated that* they see 
from two to six per year. But these suggestions do not usually result in « 

changes, since the suggestions are often general in nature, do not Jtead to 

feasible changes, or point out problems which, training developers . are already 
aware of. If possible, '.DTD personnel respond to personnel suggesting changes 
and state what change waS made or explain why no change was appropriate. The 
low rate of - .suggestions . Xroin..rhe _field...should. thus _not -be due. -t o t he a t titude - 
that they will not be attended to. Other causes may be a low usage rate of 
training materials in the field, or a hesitancy to take the time to put com- 
ments into writing. Interviewees also indicated that they receive unsolicited 
letters and phone, calls from field personnel at the rate of two or less per 
year. Field personnel for some reason do not frequently use formal channels 
to cotnmunicate feedback to training developers . >Ieetings at conferences .also 
do not contribute much feedback. Several interviewees indicated that confer- 
ences provide only general feedback which is usually not useful. During. 20 
interviews only one instance of useful feedback being obtained during a cpji- 
ference was reported. This occurred when a battalion commander at tending;; % 
conference actively sought out personnel in a branch of the DTD and provided 
specific feedback on use of specific training rakterialsVin his unit . Methods 
are needed to increase the frequency with which such feedback is provided. 
Reliance on approaches as unstructured as those described in this paragraph, 
is unlikely to produce this result. . ' \ . 

A type of informal feedback which more frequently provides useful informa- 
tion is based upon the turnover of military personnel within the. DTD. Officers 
and NCO's transferring into tfe DTD often come from an assignment in Jie fiettd. 
They thus bring with them .useful information on the, quality of soldiers^and 
use of training materials in the field. This is. a benefit of the hightnrbu- 
lence rate in the military assignment system and is a way to increase pongruence 
between training developers and field personnel. However, the, information . 
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gathered by this method Is limited by human pemory-^and by the experiences which 
training developers had while assigned to field units; This type of feedback 
is an important one, but it should be supplemented by information gathered 
regularly in a wide variety of units. 

Z\ Internal feedback . The types of feedback described thusfar in this 
section are* external dries relating to soldier performance and use of training 
materials in the field. Training developers also redeiye internal feedback * 
reiating to institutional training from other directorates and departments in 
the Center/School. A large amount of such feedback is based upon informal 
exchange of Information between personnel in OTD and other agencies , but 
several A more*formai types of internal feedback are available . Instruct ional 

departments submit. -.r^uas-ts-- for --^hange^ — to — 

training developers. Such requests include a rationale for why changes-are 

heeded - r and thus provide feedback on where problems lie in trainings Student 

c r ITIq ue " sTiee C so r io £Ke r~ "MUtvfey S~ a r e u su^l 1 y c otnp leted at t he^c one 1 us ion of 
courses, and results are available as feedback b the DTD through the DOES, as 
desc rTb"ed~~earITe r . To we ve ir , se ve r al p f the DTD personnel interviewed stated 
- that- they- do -not* -consider- -this bu : : ±mportOTt : : f ornr of- ■ feed-back; ■ dud' to- concerns 
about the validity of s'tudents 1 comments, the general feeling is that students 1 
comments are influenced more by the personality or presentation style of the 

instructor than by the content of the course . Training developers are more 

.likely to take stock in comments received from instructors than in those received 
from students. . : 

. Several procedures. followed by training developers during design pfs training 
j)rpgr^ms . aja<J in&k«i3^ diejicxth^ 

Experts on various tasks at the Center/School are often. Informally sought out arid 
consulted during training development. .For example, if training on a chemical 
task is Included in a training program,, local experts on such a task will be 

id e n 1 1 f i ed ,. 'an d_a sked_ t o _r e v lew . „the task do cumen tat ip.iL, Mt e_r_„a_^ t^_ini^g_^pj^r am_ 

is initially developed, it may be pilot tested through student trials or pilot 
teachings. Student trials involve administration of a part of the training - 
program to a small group of students available on a temporary basis. Pilot 
teachings are the first adrainist ration of a new or revised training program to a 
regular group t>f students. DTD personnel are directly involved in both of these 
procedures and usefui feedback on the adequacy of course design is gathered 
through observation and interviewing of students. The information gathered falls^ 
under the general mbilc of f eedbac k , al l UougU i t-^retetes-t o the evaluation- of- - 
initial or dr^ft products rather than finished products. DTD personnel seldom 
have the opportunity to dbserve the administration of a training program af£er • 
the pilot teaching. Such observations are performed by DOES personnel, and re- 
ports of- results have in the past been provided* tb DJD on a quarterly basis (as 
mentioned in a previous section, .such, reports are not currently being provided 
to DTD). >fcst interviewees indicated that they receive such reports and find 
them useful; 'the results of performance tests conducted during training were 
frequently pointed out as being especially useful information. * 

Summary. Several types of feedback are currently available to DTD person- 
nel and most of these have been described above and wer£ previously summarized 
'in Table 1. 1 Other types of information are available which could be construed 



as feedbacks but do not merit separate discussion. For example, training 
developers receive commiTits dtirihj; staff int; o f t rn ining programs and mater lain y 
with Center/School and TRADOC agencies, , Itpwever f such comments do not fit the 
strict definition of feedback since tho'y rotate to draft products rather than ? 
to finished products that have been used in the field. Also, personnel inter- 
viewed in DTD indicated that generally few comments are received during staffing 
of training materials. Specific survey and data collection forms used by the 
DOES to gather feedback were described in an earlier section of this report and 
have not been further discussed here. For example, DOES personnel administer a 
survey to field. personnel on the use of ARTEP materials arid provide the results 
directly to ARTEP developers in DTD. The intent of this section was to provide 
a general description of feedback available to serve as a basis for detailed 
discussion of f eedhack needs below. • " 

While many of the types of feedback available, to training developers are 
"very useful, several problems are apparent in the f low t of feedback information. 
In general, the- flow of feedback appears to be informal, unsystematic, and 



based largely upon " suBXec t ive data. Key issuesand nce^dr are identified and 

discussed ' across types of" feedback bclowv .-fir order ;tb- develop recommendations- 

for improving feedback flow. 

Major Feedback Needs 

Lack of feedbaefc . The most general conclusion about the feedback avail- 
able to training developers is that there isn't enough of it. All 20 inter- .- 
vie wees who indicated a need for feedback from the field also indicated that 
they present -receive that, they as . 

training developers should' be more directly^ involved in gathering feedback and 
should have more direct contact with personnel in the field. Changes occur in 
the field rat a rapid pace and training developers have difficulty keeping in- 
formed of them. Problems are often surfaced through informal channels or 
rumors and training developers .may not have sufficient time to react to them. . 

Interviewees did not present a consensus Jm how, the flow of feedback . 
should be* increased. ' Some stated that they should visit field units and provide 
assistance in implementation of new training programs and materials. Others 
indicated that increased travel to the field is impractical and that^ field per- 
sonnel should somehow be further encouraged to' contact t raining ^ e ^ el ^ ers / nr ^ ; 
misht be more cost-effective to bring experienced field personnel, to the tenter/ 
School than to send DTD personnel to the field. Field personnel te ^°^^ 
assigned to the Center /School to attend a training course could also be further 
used as a feedback source than they presently are. These individuals could be 
surveyed or interviewed about specific areas of concern upon their arrival at 
the Center /School. As described earlier, these personnel currently complete a 
general survey during their first two weeks at the Center/School but ey are 
not interviewed in depth. Several interviewees expressed the opinion that feed- 
back on training materials-is rarely received from the f ^ ld ^ becauS ^^ fioaia 
materials are xa fact seldom used. '-The first step in gathering f eed ^^ r ^°f d 
thus be to .survey the field on the availability and use of training materials 
Se same argument can be made for training devices. : These and other suggestions 
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for" -tncr?n.?tng- the amount of feedback available ro tte DTD will ho used to 
develop more spec if ic* rccommendat tons in a Inter sect ion of thi& report; 



Specificity of fe edback . A c o fnmo n o h kc r y a t to h o f f er e d ; by a 1 mo s t nil 
interviewees is that much of the feedback .available is not suf ftc icntly 
detailed or specific. A general problSm may bo identified* but* details are 
usually not provided which isolate thejexact cause of the problem and ways to. 
correct it. Th}s situation exists with trip reports received from DOES arid 
other sources and with roost of the info rnva L -feedback received. If field 
personnel indicate that graduates of institutional straining are not well 
trained* or that training materials are not useful , attaining developers need 
-f art4i^r- in | ornMtt ~ iwr 7 Thcy^reed to~-fcnow~wtrirfr^^^s^ 

causing performance difficulties and which parts of training materials are 
not being used arid why. As . in the example used earlier^ training developers 
need to know not drily that lieutenants can't read maps, but which lieutenants 
have difficult y wi th wh ich a s_p_e c ts b f map reading/ Training programs can. _ . 
then be redesigned to attempt £0 remedy the problem. *■ vC. 



DTD personnel rely heavily upon DOES and other- agencies to gather feed- 
back for them, or upon field personnel tq_ yplunt_eer feedback. This, leads to 
v the provision .of feedback without "so* isfactory detail. It is often tedious; 
arid tiroes-consuming for training developers to go back to the source of the 
feedback and obtain farther information. One solution to this problem would 
be for these personnel to be directly involved in the original gathering of 
feedback'. Further details could then be gathered^direct ly from the feedback 
source in one qperat ion . Another solut ion would be to increase the coordina- 
tion between those who need the feedback in the DTD arid those who gather it 
r in the DOES . DTD personnel have the opportunity to submit specif ic quest lone 
to DOES personnel bjfore field units are visited, botjthis coord tnat ion., is 
apparently riot sufficient: Training. developed s raay^not know what questions to 
ask, since they have ^riot received previous_.f eedback as to what the primary 4 
concerns of the field are: Interviewees indicated that in the past they 
have not received answers Jo questions submitted within the needed time frame 
or at all. This perceived lack of response does not * encourage s ubmi ss io rT of '"V 
future quest ions. Even "if DOES personnel do ask questions tieveloped by DTD 
personnel in the field, they may not: have sufficient' experience with the par- 
ticular document or device being addressed to get the needed details by asking 
appropriate follow-on questions. The only solution to^his dilemma would be 
for training developers to !. accompany training evaluators on visits to field 
Units . : , , ■ ■ . 

Validity of feedback . Several of. the training developers interviewed > 
expressed concerns about whether the information they receive as feedback is 
: valid and reliable. It is frequently difficult to determine how representative 
. feedback is of the situation* in. the field as a whole. For example,, a problem 
with a particular document may be identified through comments in a Jtrip report 
of through informal means - r * such as a phone call from an officer in the field or 
a comment ' from an NCO newly assigned to the DTD. Is this' problem widespread 
throughout field units or does it represent the experiences of one individual 
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in just one tmlt?. -tf»eiee I3 : at. p?«?5tHie h<5 : »cructiire^ v .wft^_ : "-tb : «nswer*vthld -^ues^-;-" 
• t Ion . When a jirobl viii _ La iMji^rtt I I"1L»U\ .inotlio^sj ai^e _iic«<it*-^'t ; o pursue' it in " ' ^ 
future collect ion of f eedbac A. tn order to d«\erminc its- extent. Clear criteria 
sUould be -eatahl Lnhed 2-9 to' 'which and how many pofSbririil should, be surveyed ot^ 
5 interviewed dlirihg \ f leld vis iit s, and as ' to .haw many times a problem should 
T surf ace' before. It is ; considered severe cridM.fth.to> require action. ^Formal and 
informal means of fe^foack collect Lon should be coordinated so that informal 
feed-back provides input ' into areas to be* further pursued by more formal 
methods. • 

Much of the feed'back'-presently received by training developers is based 
upon sob j e ct tve- opJniQn s gi personnel in the f leHv --- This-- i-s true - ^rtforaa^ r 
tion gathered jthrough trip reports, C.ODAP surveys, and -informal methods'. Many 
interviewees expressed coniem with the validity of such* information and indt-< 
catcd .a need 1 fpr more feed-back based upon performance testing'and observation 
of - field- performance. However, several of, them also expressed reservations 
about the validity of field o b s e r va t i on s , . ba»se d on the feeling that field! units 
only let ,you see what they want yq^i to" sec. Few performance test results i are • • 
presently available, and fewer, wii 1 ' be' available in the near future. SQt re- 
sults arid results, of. te*tiL given . dur ing..inst.i^t,lpaal.. training represent the 
only performance results available as. feedback at* present . The hands-on 
performance tes,t- portion of the-SQT is scheduled to go by the wayside in- the 
next few months,' so no performance results will be available as external .feed- 
back at that time. # 'All these facts /nuike it difficult to alleviate, training 
developers* concerns about the validity of feedback. 

The only realistic way ,to. increase the validity of feedback Is to base it 
further upon observations in the fie tdV : Observers must be trained to make r 
thorough arid abjective observations which are not unduly influenced by opinions 
arid interpretations: Observers should *lso present themselves as assistants 
arid facilitators 'to the field, rather than as evaluators v Useful feedback can 
then be gathered through observations'*. The only "other way to increase the 
availability of valid feedback would be to increase the performance testing of 
' f±e 1 d personnel 1 »* this'" alt erna t fve is' cb n s Ide re d an realist i'cT; ' s ihce. the Ariny" 
is currently moving away from the' testing of individual .skills in the fie^d. 
However, if the validity of feedSack is* ever to be maximized-, -it will have to 
be done through .performance testing. <• " * 

'■ '■■ ■ * : .', - ■ °. ■? - ' ' ^ _' • 

• Timeliness of feedback ' An6t Her Imajo'r; problem that surfaced during, the 
interviews is that many types of feed^acjc are- not timely . By the time that 
' information is received, the situation, in 'the field may have changed and it 
mky be too late for training de?elbper^ 

ro by before information is summarized in trip *e P orts and distributed to those 
Who need it. One interviewee stated that it has in, the past taken, as long as 
eight months to receive answers '. t p questions asked in. the field through the - 
DOES. In this time fxame the -question designer tnav.have been transferred or 
may no longer have need for the feedback, since he or she was required to 
complete actions without. it. CODA* surveys ?f enlisted- tasks within, a par- . 
ticular MOS are conducted every JfKitS year,, and purveys of officers ***** 
have thusfar been W-time fecial exercises. -Task Analyses can thus be- based. 
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on information that is as much as three years old , ^ Given t.il^ rapidly changing 
state of technology in today's Army , training materials may Be out-of-date v^heti 
they are developed. Item analyses of St^V results are another example -of im- 
;.£itiB&ly. Jfeedba.ck._. These analyses have in the past riot Been received.- in tinie to 
Be used" iii 'development of. the next iteration of tests. 

If, as suggested earlier, training developers were mpre directly involved 
in the collection of feedback, its timelinegs should improve. Direct contact 
w'it'h the field Would allow training develope^rs .to_^ determine whether the informa- 
tion they are, using ,is current. Ifore freqaent afe^istrati^ of; CODAP 'surveys 
would also provide more up-to-date infprmation to training- developers;, ,.■ If 
this is not possible, training developers should be provided resources- to visit 
the field themselves arid supplement CODAP. information as ridded. , Feedback^ will 
not be useful unless steps such as these are taken to improve its timeliness. 

Continaity of feedback . Several "interviewees made; the comment that feed- 
back needs to be more continuous than it presently is. An example of the -non - 
continuity of feedback is the administration of /■■CODAP surveys discussed above. 
These surveys are conducted every three years and. no updated information is 
gathered between administrations.. This leads to the lack of timely information v 
discussed above, but problems with the. continuity of feedback are more extensive 
than that, different types o-f .feedback are often collected in isolation 'and are 
not integrated with oth^r types. For example, if a problem in .the field is noted 
in a trip report or in informal feedback, it should be further ^pursued in future 
field visits and interviews. At present there is no system for collating and 
integrating feedback and determining what its major implications are. No one 
person or group at the Center/ School integrates feedback previously gathered from 
all sources and determines what issues should be addressed in future collections 
of feedback. If a problem is identified during feedback, there is no mechanism . 
to follow it up and insure that it is. solved. Much of this situation is due to a 
lack of capability to analyze and maintain feedback information over time. This 
problem is further discussed below. : ■ 

-7Anelysis and management of feedback. DTD personnel presently have* no atko T \ 
maieS ca-pability to analyze arid-ma intain the feedback they receive. In some m 
cafces''other agencies perform a descriptive analysis of data before providing . 
the* tp the DTD. For example, computer print-outs pf descriptive analyses of 
CODAI* arid SQT results are available. But in other cases, such as data gathered 
during field visits or through-some of the surveys administered by^DOES person- 
nel - the only available method of analysis is by hand. This is a tedious pro- 
cess and the result is that data are/ sometimes, not thoroughly analyzed and used. 
For example, 'completed surveys are sometimes glanced at and thrown in a file 
drawer without ever being analyzed. The lack of an automated data storage and 
retrieval capability leads to lack of capability to establish trends in feedback 
over/ time. Different types of feedback are thus not used in an integrated and 
continuous fashion, as was discussed above. ( ■ 1- * 

If procedures are developed to increase the flow of feedback to trailing de- 
velopers, procedures must , simultaneously be developed to anal y^ and maintain the 
data acquired. An automated system is ne.e<^ Vhich can be jointly accessed by 
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collectors, of feedback. in the DQES and users in the DTD. Existing data manage- 
•,ment packages can probably be modified for this purpose, ,h.B.iit several decisions 
aioutf-'system ' design neeid to be made^ . such as -what, data are : to -be eiitered into 
the system, 'how data are to'; be displayed in a sf triple, useful manner, and what 
criteria should be established to determine when data indicate a significant 
problem in the field. Many of the problems With feedback discussed in this 
paper cannot be solved without development of an appropriate data analysis 
and management system. 

Flow d.f feedback. A final problem with feedback, and one which can be 
directly related to lack -of a data management system, is the fact that feedback 
is not" aiw4y$; available 'to those who need it. Several .of , the training developers 
interviewed expressed the suspicion "that they do no't receive all the available 
feedback which might be of interest to them. The circulation of trip reports 
appears to be rather spotty. Some interviewee^ -indicated that they see DOES 
trip reports regularly, while others indicated 'that tney had not seen one in 
•over a year. SQT item- analyses are available to SQT developers, but other 
training developers -indicated, that these results are not readily available to 
them. Results of surveys on" the use of 'particular training materials have in 
the past been available to writers of the materials but. not to task analysts 
and others involved in the production of the materials. So feedback does not 
always flow to --all those who feel a need- for it . 

Several interviewees attributed feedback flow problems to the current organ- 
izational structure of the DTD. At present,, the task analysts, writers of train- 
ing materials, and producers of* final copies of training materials are organized ; 
fiito different divisions of the DTD. And the evaluators of training materials 
are located in another directorate, the DOES. Such an organizational structure 

sometimes leads to' communications problems. For example, task analysts who 

determine what tasks are to be taught in the institution do not always have con- 
tinuous interaction with personnel who develop the training programs for these 
tasks. One suggestion which was offered by two. interviewees to alleviate this 
problem is to reor&nize the DTD by MOS. A given group would then have total 
responsibility for developing training for that MOS, from task analysis through 
training evaluation. It. is not within ^he purview of this paper to suggest a 
total reorganization of the DTD. However, some action is needed to increase the 
interaction among the divisions of the DTP and between the DTD and the DOES. 

Summary. The major problems with feedback identified by the DTD personnel 
intervSfare that there is not enough of it, it is often hot specific or timely 
enough it does not flow to all those who need it in a continuous fashion, its 
validity is frequently undetermined, and there are no efficient ways to analyze 
''and manage it. Ve key *o meeting the feedback needs of DTD personnel appears to 
be the rlgular collection of valid objective data which can be analyzed and main- 
tained so that they are easily accessible 'to all potential users Meeting this 
eoa l will requires the expenditure of more resources than are now devoted to the 
collect!^ and management of feedback. In the next section the conclusions 
reached from interviews of DTD and DOES personnel are integrated and mo,e spe- 
cific reco^eSati*»s are developed for efficient use of feedback resources. 
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SECTION V 
CONCLUSIONS AND RE COMMEND ATI ON S 



While some useful feedback is presently available to training developers 
and evaluators, the interview results discussed, in the previous two sections 
suggest that there are. many ways in wliich the flow of feedback information 
heeds to* be improved. Conclusions about present feedback methods and proce- 
dures are discussed iri this . section, along with recommendations on how to 
improve them. The recommendations are general ones which should apply to any 
eenter/School, and not just to the one _ J - n _ wn i c ^i n ^ er Y^? w ?_ were conducted; 
While it is necessary to discuss the recommendations separately here, they are 
interrelated and should be implemented in an integrated fashion. 

- - * i 

Feedback Needs • 



The most general conclusion that can be reached about the present feed- 
back system is that more feedback is, needed, This conclusion is based primarily 
upon the perspective of training developers in the DTD, who almost unanimously 
stated that they are not .getting all the feedback they need for improving 
training programs and materials, training evaluators in the DOES did not per- 
ceive a widespread need for more feedback, and some of them indicated that they 
are already receiving mare feedback than they can handle. *The users i of feedback 
want more of it and the primary providers of feedback feel that they are provid- 
ing all they can, given their current resources. Possible solutions to this 
dilemma, include involving training developers more directly in the collection of 
feedback and increasing the coordination between training developers and evalu- 
ators. These and other approaches to increasing the feedback available are 
discussed. liter in this section. - 

The paucity of feedback available to training developers in some ^areas 
indicates that they are correct in their assertion that more feedback _is needed. 
In some cases "feedback is_ needed where it is not currently available, and in 
other cases feedback is already available, but, more specif ic information is 
needed. The types of feedback needed % that are not currently available include 
objective performance results and information on the use of training materials. 
Several of the training developers interviewed* s-ta ted a need for more objective 
performance results, such as results of hands-on tests of individual skills and *. 
observations of unit exercises. Few such data are currently available; the only 
types identified during the interviews were results of the hands-on portion 6f 
SQT's and results of, institutional testing included in quarterly reports pre- 
pared by the DOES. Hands-on SQT results will likely sopn no longer be available, 
and indications are that institutional test results will no longer be reported by 
the DOES In a regular formal fashion. Training developers will thus be receiving 
fewer objective performance results rather than more, Possible ways to reverse 
this trend wilt be addressed in a later discussion of the validity of^ feedback. , 

v "toother pressing need for feedback that is not currently being satisfied^ 
is the requirement for more feedback from the field on the availability and use 
of training materials. Nearly all DTD personnel interviewed indicated a need 
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for increased feedback on the use of ^/training materials they produce. Inter- 
views with DOES personnel indicated that, wftile some of them are aware of this 
need, very little has' been done "to address it. As a first step, DTD personnel 
with DOES assistance could devise a generic set of questions that could be 
used to address a wide variety of training materials. Frojn this pool of ques- 
tions, training developers could select and modify appropriate questions to _ 
determine the availability and use of specific training materials in the _fie°l^. 
As mentioned earlier^ the scarcity of feedback on training .materials may be due 
to the fact that they are not received or used in the field. These general 
issues should be addressed first, followed by more specific questions on the 
design of training materials. In this way, training developers would be more 
likely to obtain the information they heed to revise or upgrade the training 
materials they produce. < 

The biggest problem with the feedback currently available from training % 
evaluators and other sources is that it tends to be too general for training 
developers' needs. Statements to the effect that lieutenants cannot read maps 
or mechanics cannot troubleshoot are worth very little to training developers. 
To'change the training given at the Center/School in map reading or trouble- 
shooting, more information is needed. As discussed earlier, training develop- 
ers need to know which soldiers are having difficulty with ^hich, aspects of the 
task of concern. "They also need to know the size of the sample and the methods 
used in^da^ta collection, so that the extent of the problem can be determined. 

There are several ways in which the, specif icity r of feedback „can be in- 
creased, all of which involve increasing the interaction between DTD and DOES 
personnel. So that more specific questions might be developed, DOES personnel 
should "extensively" coordinate their data collection plans and forms with DTD 
personnel before making field trips. The inclusion of DTD personnel in data 
collection teams would make available knowledgeable personnel who could ask 
specific follow-on questions during interviews. A more continuous feedback 
system than presently exists would provide for maintenance of data^axJ^fss^ 
field visits-. General problem areas uncovered in one visit* could then be ad- 
dressed in more specific detail in later visits. These possibilities are fur-' 
t;her addressed in the discussion below of coordination between training develop 
ers amd evaluators . . 

- Coordination Between Training Developers A nd^E valuators 

A properly functioning feedback system can be described as a continuous 
loop. When training developers and evaluators are not the same people, as is 
the case in most TRADOC Centers/ Schools , continuous coordination is required 
along this loop. Developers of training programs and materials should provide 
input to training evaluators as to what feedback is needed. The evaluators 
should then collect the needed information and provide it to the developers, 
along with at least preliminary analysis and interpretation of the data. De- 
velopers should then inform evaluators as to what was done with the feedback, 
what changes were made in training programs and materials, and what new feed- 
back is needed. Continuance of this cycle should insure .t.tvat training devel- . 
opers receive the feedback that they need. Unfortunately the interview 
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results indicate that the feedback process does not operate through a contin- 
uous cycle in the real world ; • 

i . . ■ . ■ —• • 

. _;_ %. ..._;_» .... _^ ?. 

r> The interviews revealed many examples of the. need for increased coordina- 
tion between training developers and evaluators. DTD personnel appeared to 
be unaware of many of the .types of feedback that are collected by the DOES arid 
several of them expressed a hesitancy to accept as valid any data with which 
they were not involved in collecting. DOES personnel appeared to be unaware 
of the types of feedback heeded by the DTD, and in many cases they did hot know 
how, or even if, DTD used the feedback that they provided . As a more specific 
example , DOES personnel obtain subj ective estimates of task performance pro- ' 
ficiency, frequency, and critical it y during graduate/ supervisor surveys admin- 
istered in the field and feedback seminars conducted at . the Center/School. DTD 
personnel who have a need for such data did not indicate an awareness. of what 
the DOES collects^ and they sometimes' visit field units. to collect similar data 
themselves . As a result , the similar sort's of data collected through graduate/ 
supervisor surveys, 'feedback seminars, \visits of DTD personnel to the field, and 
CODAP surveys are not integrated. 

There are several ways in which the coordination between training developers 
arid evaluators ca|v be increased. DOES per sbnnel should increase their efforts 
to involve the DTD in all phases of the feedback process, ^particuiarly in the^ 
development of questions and methods to be useti in obtaining *data from the field. 
As was suggested earlier , DTD per sonnel could develop pools jof questions for 
assessing the availability and use of training materials. The DTD or instruc- 
t ional departments could develop pools of questions for other purposes, such, as 
assessing the performance of soldiers or the distribution and use of training 
devices. , As soon as the DOES has a tentative schedule for the^ units to be 
visit ed diiririg a given quarter, the schedule should be shared with the, DTD, sp 
.that DTD personnel have ample time to add non-standard items of current interest 
to the' data collection plan . DOES personnel presently coordinate their field 
visits with the DTD and other departments, but the present informal system of 
sending a request to the' DTD for questions ; just prior to a field visit appears 
to be inadequate. If 'DTD , personnel have hot received previous feedback from the 
DOES, they may not know the appropriate questions to ask. The development of 
pools of questions as suggested above might help here, since questions could be. 
selected from these pools rather than being developed anew before each set of 
field visits. It may be necessary for the DOES to establish points of contact 
at the action officer level in the DTD and departments through whichfield 
visitations would be announced, data collection needs > would be tranjijfct ted, and 
responses to these needs would be returned. ' personnel could alWmeet with 

DTD and departmental personnel prior to a "visit to map out what they expert, to 
accomplish during the visit/ This involvement of training developers in data 
collection could go so far as to include these personnel in teams visiting the 
field; this possibility is further discusserd below. Meetings between DOES and 
DTD personnel following as well as prior, to a* visit $hould also be beneficial. 

To insure that feedback is useful to those to whom *it . is provided, both 
training evaluators and developers should take steps to'close Che feedback 
loop. The preceding paragraph discussed the necessity .for. training developers 
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4 - m input. into the design of feec£ljack questions* arip methods, . In this .way the data 
heeded by training de^eibp^rs ate more likely'*to ; be gathered. Jhf _^ES_jroiat^_-_ 
then provide feedback to the- DTD on each quests ion asked aiid on reasons why any 
requested data w^re hot collected: Where specific problems are, identified dur- 
- irig an on-site •Visit', training evaiuarors 5 sfio old provide recommendations on how. 
these problems might Be addressed Although -such recommendations need hot be 
binding upon training developers, these personnel should always inform the 
training evaiaators of-actions taken to address the problems-identified If 
the problem is not adequately documented to^enable the training developer to 
address it, the training evaluate r should/oe so advised. In these ways useful 
feedback can be obtained on a continuing basis.-, Ah example of a feedback sys- 
tem which normally provides for periodic coordination between training develop- 
ers and e Valuators is the Missile and Munitions Evaluation conducted by the 
Missile and Munitions Center/ School (MAME, 19&2) . r ■ 

i * 
Field Con tacts Fdr Training Developers! \ 

Training developers develop their products in«-the relative isolation of 
U h'e .Center/School. While some training developer^ have gained extensive field 
experience in previous assignments, .their knowledge is" rapidly outdated by 
changes in doctrine, equipment, and the 'threat. Some training developers in- 
dicated during interviews that ' they must constantly struggle to keep up with 
changes in the -field. To provide product sf that; -are, responsive to the field s 
needs, DTD personnel, must increase and- maintain t-heij direct contacts with 
field personnel. Two approaches to this - ar«^suggested below. 

Interviews with field personnel, at ; the Center/School. 'Perhaps the most 
available and underutilized jfesource.'Eor obtaining, external feedback is field 
personnel who return to the Center/ School for training in advanced NCO or of fi- , 
cer courses: With their experiences in the field still fresh in their minds - 
and the distractions of their field duties temporarily removed, they can provide 
much useful feedback to the Center/School.- These soldiers should be interviewed 
individually by DOES and DTD personnel shortly after .their arrival at the- 
' Center/School. Structured interview formats should be used in sessions requir- 
ing no more than an hour of., the interviewee's time. Interviews could cover 
practically any subjedfc matter than can be asked of the same soldiers in the 
field and need not be restricted to perceptions, of graduate performance, as is' . 
' presently done. °In fact, if might be instructive to ask them the same questions 
that are asked of their counterparts in the field to determine if ^-feedback 
Obtained is comparable. :If the information obtained from soldiers returning to 
^he Center/School varies' little from that obtained in the field, then the number 
f of personnel interviewed during field visits could be reduced. Interviews con- 
ducted With field- personnel at the Center/School are more convenient and less • 
■ expensive than thesame interviews conducted in the field. WD P«"^J «£f fc 
also construct short questionnaires to answer spec if xc questions of immediate 
con cern about their products that are not addressed in field, interviews or 
questionnaires. Administering these questionnaires to field personnel at the . r < 
institution would provide quick and cost-effective feedback. Field personnel. 
temporarily located at the Center/School are a feedback source thatshould be , 
utilized more extensively; * • i\ "4 ■ 
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* Field visits . Another way in which contacts between training developers 
"ahd field personnel could be irie teased would be to include training develop- 
merit personnel on the evaluation teams that routinely visit units in the; field; 
^Although the DOES occasionally includes personnel from other directorates/ 
departments on its external evaluation team, the number of* times that trailing 
developers have been included is far too few; The DOES should ; include at least 
one training developer -on every field visit. s . ; 

Training developers should, accompany DOES personnel as active observers • 
Although they would not be the key data collectors, the training developers 
might be involved in administering questionnaires, observing unit operations, 
and conducting interviews with lower ranking officers, NCO's~ and; enlisted ; ,v 
"personnel; It is important for training developers visiting field units >to have 
a definite but flexible agenda. Prior to the tr£p questionnaires and structured 
interviews should be developed and plans made for collecting the data and making 
observations. Several training developers expressed concern dur ing the inter- 
views that they would be allowed to see only what units want them to se~e in the 
field. Data collectors|»should be trained to get around .this problem by looking 
for objective indicators of unit performance. Much can be. learned through ob- 
jective structured observation of unit activities ; this point is further ad- 
dressed in a discussion of the validity of feedback below • ' ; " 

; : " : ' ' . ' : _ V > - \ _ 

Iri addition to, providing badly< : needed contadt with field personnel i_there . 
are other advantages, to including training developers on- field visits. Training 
developers are likely .to have Sore' confidence )in feedback that they themselves^ 
have been involved in collecting. They will Jbe more motivated po develap medlfed 
for obtaining the kinds of feedback they neefl, sincf . they know that the needed 
information will be collected. They may also come tp better appreciate' the 
difficulty of obtaining useful feedback from the field. Also, as suggested ear- 
lier, training developets might be, able ' to develop c%ailed follow-up questions 
during- interviews in the field,, due to their .direct; experience with the products 
being evaluated. * * J " - 

There are, however, at least two potential problems. with sending- training 
developers to collect feedback in the/field .__ The first is- the ^ additional cost 
of sending additional people to the field. However, ^ftis procedure might save 
ioney in the long run if it leads to improved trailing -materials and better 
trained soldiers. It might also lessen' the numSer o£ 5 training evaluators * . 
visiting the field and reduce the need for training developers to conduct, sep- 
arate visits on their own. The second pr^SLem is tjiat training developers ,__ 
might introduce" bias into the data they collect. They^ might inadvertently, or 
purposely, make their products (i.e., training materials and .trained. soldiers) 
look better than they actually are by asking questions in a biased manner or 
by not being objective during interviews. For this reason training; developers, 
should receive instruction in objective methods of data collection., before they 
visit the" field, arid the data they collect should be used .only internally for 
identifying problems and correcting deficiencies and hot for more general eval- 
uative purposes . v • * 



Validity Of Feedback 

._ Subjective versus objective measures . Much of the feedback presently 
available" to training developers is subjective ih that it based on the 
perceptions arid opinions of unit personnel. Among the motfe subjective data 
are responses to student questionnaires, ratings of graduate per fbrmarice, CODAP 

survey results, and pro f e s s ibrial ism survey results . If sjfejective -measures 

like these are going to be used* then their accuracy or delidity must be demon- 
:"Strated by comparing them pith more objective measures, Such as results of tests 
arid observat ions. If the subj ectrive measures do not comf are favorably^ with the. 
objective dries,- then the subjective measures must be replaced by more accurate 
ones. „ 

Ratings and other estimates of graduate proficiency on specific tasks^ such 
as those .collected by the DOES and in "CODAP survey s,~ have not been generally 
shown: to be accurate when compared with more objective performance measures 
(Burnside, 1982) . Therefore, DOES J personnel • should compare their subj ective 
measures of task performance with the results of objective performance meas- 
ures such as hands-on tests. Unless a significant relationship can be demon- 
- strated between performance estimates and hands-on performance, the former meas- 
• ures will have to be abandoned for more accurate but expensive performance 
, measures. 

. ■ " : % . _ • /_ _ _ 

' - . The : accuracy of estimates of task criticality and frequency of performance 
as cgliected by the DOES arid in CODAP surveys ; has also not been demonstrated. 
Butnside (1982) found few studies supporting the accuracy of either frequency 
orrttiticality estimates. Estimates of time spent on "specific tasks arid fre- 
' ? ' quency of performing these tasks are generally inflated, leading to the neces- 
sity to use relative rather than absolute judgments. ,To determine' how accurate 
subjective measures of frequency of task performance are, the DOES should ob- 
serve the frequency' of performance of ;a selected set of tasks in a field setting 
arid compare this objective measure of frequency with frequency, est iraates. Esti- 
mates of task criticality are notoriously unreliable; If task criticality esti- 
mates are to be used at all, -special techniques- may be required to obtain them. 
When soldiers are asked to rate criticality, almost all tasks are rated at the 
high end of the scale. That is, almost all tasks are considered critical. This 
problem can be overcome by forcing soldiers to . rank .order the tasks from most to 
least critical, but this becomes quite difficult as the number of tasks in- 
creases. Burnside (1982) discussed another method for obtaining criticality 
estimates whereby the critically of each task is compared with that of .each • 
and- every other' t as* in pairs. Although 'this paired comparison teen ng^as_ :- : 
been shown to increase, the reliability of criticality estimates it ^comjs im- 
practical when the number, of tasks is more than six or eight : Became _ 
liability -of criticality estimates 'is typically low, their use ±S no J "cdm- 
Vmended for feedback • purpos es unless their reliability can Ue improved by using 
special data collection techniques. > * - ; 

To be- useful, measures- of training effectives'^ 
: accurate •information so that 'deficiencies in the- training ; can be • identified and 
•;|orStiiS BeKuse their penary goal is not to evaluate training, students may 



^ not _make _ the detailed observations needed for evaluating and modifying training. 
Students 1 * responses _td questionnaires about the effectiveness of training may 
consist primarily of gross, judgments regarding, the training they received^ this 
becomes moire likely when the questionnaires ask students to rate the^ instruct ion 
of instructor on each of several dimensions. When student questionnaires require 
such ratings, they are subject to all the problems associated with ratings men— 
tioned earlier. In addition, responses to student questionnaires may be affected 
disproportionately by the style and charisma of the' instructor . For these teas-, 
ons student questionnaires should not be used ap the; primary measure .of training 
effectiveness. Generally training effectiveness is more accurately measured by 
sending carefully selected, weii-trained observers »to objectively record their 
observations of training on structured observation forms,. The DOES presently 
sends observers, to inspect institutional training but structured observation 
forms are hot always used - y and thos£ forms tnat are used depend too heavily oh 
subjective assessment of the training. Objective "observatibri'bf training coup- - 
led with rigorous erid-of-blbck tests is recommended as the best way to evaluate * 

. training. Both training developers arid evaluatorS should apply this approach in 
both institutional and unit training^ draining e^aJ-uiitions should be conducted 
in accordance with guidelines provided in ARI Research Products 81-15 through 
81-18 (Kristiansen, 1981; Kristiansen and Witmer, 1982a; Kristiansen and Witmer, 
19826; ' Witmer, 1981}. 1 ' ; K - ' 

» . — . * < 

Most of tfhe items on the professionalism ^survey caTl for- subjective re- 
sponses. Almost every ite^i calls for either an attitude, an opinion, or an eval- 
uation of some aspect of the Army as a profession. The -purpose of the profes- 
sionalism survey is to dderitify attitudes i of opinions thart may relate to person- 
nel problems such as officer or NCO attrition. However, the DOES has made no 
effort to compare survey results with actual attrition lfates, and, as best as 
could be determined, the validity of professionalism surveys has^ never been es- 
tablished. Furthermore, DOES personnel interviewed indicated thkt these surveys 
are of little value in identifying personnel problems. Unless the validity and 

' utility of pro fessionalism or similar surveys can be demonstrated, they should 
not ..be msed for feedback purposes. 

* Even relatively objective measures such as tests and objective question- 
naires do not always provide valid feedback. For example, tests may be admin- 
istered in which the soldiers are given assistance or re not tested to the cor- 
rect standards. On questionnaires, the questions may jV worded such that the 
respondent is biased toward giving the answer that theHuastioner prefers. 
Questions asked* during interviews may also^be worded so* that the interviewee 
• gives 1 the desired answer, and interviewers may influence the interviewee by the. 
way that they react to the answers provided. To inc rease the validity of data 
gathered through questionnaires and interviews, DOES and DTD personnel should 
carefully develop the questions^ to be asked in advance, paying close attention 
to the wording of the questions. Questions should be aimed at obtaining factual 
information rather than attitudes or opinions, and should only be asked of those 
soldiers who are likely to have the relevant experiences to provide the informal 
tion requested. Questionnaires arid ' st ructured interviews developed for feed- 
back purpo ses must not only provide information that is valid but also informa- 
tion that is useful. Therefore, the questionnaire or interview developer must 



take special care to include only those questions that yield inf.br mat ibh having 
a clearly identified use for a specific directorate/department , For additional 
guidance in construct ing quest ip.nnaires and structured interviews see .Dyer - t 
Josephine, Wright, and Yudowitch (1976) or Kr ist iansen - arid Witmer (1982*1* Append ' 
dix 5) . 

./•i ' ' V .. . __ : _ ■ ; .' 

* Presently, feedback to the Center/School is based too much on subjective 
measures and not enough on objective ones . While interviews and questionnaires 
can provide valid, useful information when constructed property, some of the 
questions now fcsed by the* DOES and tile lack of structure in the interviews in- 
vite subjective responses. Along with efforts to improve questionnaires and 
interviews , * training developers and ^valuators should increase the use of more 
objective methods, such as observation and hands-on testing. Although thes^ 
methods are relatively costly and time-consuming,' their application to a small 
sarapie of units may very well yield more useful feedback than would subjective 
measures gathered from every unit in the Army. 

■ Criteria :; f or samples . Maintaining" the objectivity of feedback does not 
start or end with the collection of data. To insure that the data are represen- 
tative of the units visited and field units in general, the .sample of soldiers 
from whom the data are obtained must be carefully selected.. Care must be taken 

to select soldiers of different ranks and avoid getting .all feedback from the . 

senior members of units- At the present time, the DOES makes little or no effort 
to specify the size or composition of the sample to insure that it is representa- 
tive. The DOES's failure to specify clearly in advance the sampie from which 
data will be obtained not only adversely affects the general izability of the v 
data., but also reduces objectivity in dec is ion -making. . . 

Several t*f the DOES personnel interviewed expresseoVdif f iculty in deciding 
how many times' a problem should be mentioned by field personnel before it^couid 
be considered serious enough to warratt taking some action. This difficulty 
arose because the DOES ha;s not established decision-making'criteria for deciding 
when to act. Without such criteria the decision to include a problem as feed- 
back to elements of the Center/School is a subjective decision. DTD personnel 
interviewed indicated an awareness of this problem an<T» several of them expressed 
a hesitancy to accept feedback from the DOES as valid, since they did not know 
the extent of problems" identified. .Decision-making criteria are essential to 
maintaining the objectivity of feedback and they should be developed by the DOES . 
But to develop these criteria, DOES personnel need to know the size and composi- 
tion of the sample upon which, the decision will ^e based, ^^^""^ J° c ^ e 
asked of the sample, and the kinds of responses to each question that indicate 
problems which can be solved by the Center/School. While the use of decis ion- 
making criteria does not guarantee correct decisions will be made it does insure 
that consistent, objective decisions that do not vary from one decision-maker to 
another will be ma4 e . 7 ■ « s t 

/ ■ ' ■ . . 

Data Analysis Arid ManaRement System • . 

Neither the DOES nor the DTD addressed in this study has an adequate system 
for analyzing, maintaining, or distributing the feedback that they get. Althoagl, 
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both 'the DOES arid the DTD have access to computer facilities in other ideations 
at the Center/School, neither has a computer on-site that can be used for stor- 
age ah4 analysis of feedback data; The. lack of an bn-^te computer . facilrjRy 
tends to discourage thorough analysis of the data, sin&e" the analysis must 
either jbe performed by hand or through a remote computer terminal. Presently. 
DQfiS arid DTD personnel store 'the feedback they- receive in file drawers through- 
out th§ir$>f f ices. This method of storing data 'quickly becomes cumbersome with 

different data in dif ferent locations and old data ♦phoved to ,fche back of the 

file where it remains untouched until it is destroyed to make room for new data. 
As a result , trends .in feedback, over time are not established and data which- may 
be similar are rarely integrated or compared. This, leads to a lack of continu- ; 
ity in feedback^ as was discussed earlier. 

__,'._„' _ • - . V 

The DOES, as the primary data collector aiTd feedback provider at the 
Center/School, needs an on-site computer for. analyzing and maintaining the data 
collected. ' .To obtain maximum benefit -from _ the computer, procedures must be de- 
veloped for analyzing and ^maintaining : a:he data, acquired. Existing data manage- 
ment and analysis packages can 'be use^/ for • this purpose to a certain extent. 
These packages may need to be modified or supplemented by other techniques to 
fit the particular needs b£ the DOES* or those to whom the DOES provides feedback 
Prototype, data analysis and management techniques should be developed based upon 
existing daia and expanded as more data become available. These techniques may 
include object iVmetho'd^'Sor establishing and updating long-term data trends, 
procedures for displaying complex .data in simple pictorial formats, decision- 
making sub- routines based on objective criteria for automatically identifying 
critical- problems, and methods to monitor suggested revisions in Center/School 
training in order to* determine their effects . 

Care should be -taken to insure that the computer does not become just a 
depository for -large' pools of invalidated subjective data. For example, CODAP 
data are- highly subjective and may be out-of-date by the time they are available 
The Army produces a number of da.ta files, scfme of which are based an relatively 
objective information^ and others that consist primarily of subjective data. The 
J50ES should" maintain' awareneas^ of the methods used to collect various types of 
feedback data. Data obtained by objective methods such as systematic observa- 
tion or hands-on tests are usually reliable and can be entered into the data 
'base without further verification: .Subjective data on the ^^^f 'p^nd^ 
always be checked by comparing it to data obtained via more objective methods. 
For example supervisors' appraisals of subordinates' performance on specific 
tasks "e not acceptable for entry in the data base, unless these appraisals can 
be shown to strongly relate to measures of hands-on performance • SimilariX 
responses to studerit questionnaires may^not accurately measure^raining effec^ 
tiveness and should not be included in the data base unless they are shown to 
relate to systematic f bservation or other objective measures of training effec- 
~ • ~j — ~_ *. 



tiveness , 



Feedback data stored in the computer should be readily accessible by 
DTD and pther training developers as well as by the DOES. Ideally DTD and the 
instructional departments would have easy access to these data through terminals 
located at the DTD or the ihstrUfc tional departments. An appropriate command 



from a terminal to the computer would create a printout of the various types of , 
data (e.g,3 test results, responses to questionnaires) available in the data 
base arid the options arid procedures available for analyzing or display itig them; 
From this list the user could select rhe cype of data and me ^ od of analysis to > 
be • employed . Subsequent instruct ions could be used to select the time frame 
and soldier population of interest. With such a system the user could quickly 
retrieve the data needed; 

The DOES computer could be used_ to quickly generate tables, graphs, sta- 
tistics, |and summaries q£, major problems observed within a given time frame fqr 
inclusion in monthly or 'quarterly reports. . This would decrease the amount of 
time required > by DOES personnel. to produce a report, thereby increasing the . 

\<ime available fjpv -collect irig d at; a.- or conducting special studies . These reports 
could be- stored permanently in th&vcomputer and could be readily accessible to 
users through terminals . Users could also create their own reports for special 
purposes by selecting the appropriate- analysis options and obtaining a printout 
through a terminal. Direct access to feedback data and reports based on these 
data would greatly speed and increase the flow of feedback from training evalu- 

'ators to training developers. It. would also allow feedback to be tailored to 
the needs of users, resulting in closing of the continuous feedback -loop dis- 
cussed earlier. , "' 



Summary 



rov- 



Training developers are not 4 getting all the feedback they need for 
ing training programs and materials. Additional feedback is needed on use of 
: training materials and more specific feedback is needed on graduate performance, 
including information identifying which tasks and parts of tasks soldiers have 
difficulty performing and the reasons for pop;r performance; ^ In providing this 
feedback it is important for training evaluitSr^^ sample and time 

frame from which the data were' obtained and' tife methods useHor obtaining the 
data. r ; , 7 ' ' 

k one problem with the flow of feedback \s the lack of coordination ^between 

training developers, and evaluators . ; Training developers need to become more 
involved in the development o^ quest ions use dNiri; data collection instruments 
and 'in the data collection process itself. Feedback sources; available at the 
Center/School should be utilized more extensively, and training developer? 
should' accompany training evaluators on visits to field units. Training devel- 
; opers and evaluators should interact continuously in order to close the feed- 
back loop. ( * 

: Presently both the DOES and the DTD are relying too heavily. on Subjective 

data, such as proficiency, frequency, and critically estimates,; Subjective 
data should not be used unless their validity can be established-.' More era- . 
ptosis' should be placed upon relatively objective measures touch as observations 
and hands-on tests. Questionnaires and interviews should be upgraded by. care- 
fully wording each question and weeding out questions that elicit unreliable- 
subjective responses-. The sample to whom the questionnaires and interviews are 
to be. administered must be specified in advance and carefully selected. 
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Objective procedures utilizing set criteria should be used for interpreting 
data and identifying significant problems; rather than the subjective methods 
presently used: 

An automated data analysis and management system is needed to s insure that 
data collected for feedback purpo ses are thoroughly analyzed , main ta ined over 
time, and efficiently distributed to those who need them. This system should 
be physically located at the DOES facilities, but it should be accessible via 
terminals to the DTD and other training developers. With the proper choice 
of data analysis and management packages , ah automated .data management system 
could perform many useful functions, including identification of trends, 
automated dec ision-ma'kin.g, arid: the efficient generation and distribution of 
reports. The availability of valid data in such a system will go a long way 
toward making feedback the continuous interactive process that it should be. 
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INFORMATION FEEDBACK NEEDS OF CENTER/ SCHOOL ; ; 
DIRECTORATES /DEPARTMENTS : 

INTERVIEW WORKSHEET 1 

Dtr/DOT t Div/B r 

Interviewee ; ; D^ e — 

INTRODUCTION 

ii Purpose of project. •_ , > ? 

a. Design of feedback system for Centers /Schools. 

b. Feedback on both soldier performance and_use_pf_ graining methods. ^ 

c. interviewing division/branch chiefs in DTD 1 , OAFMS, and other agencies as needed. 

2. Information flow schematic 

a. Sources _j , • 

b. Division/branch processingxbf information 

v c. * Feedback from users . 

3 J Conduct of interview , 

ft. Organized by type of information ; 

* b. Not an evaluation or audit of anyone's job 
^ Want opportunity to come back later 

, INITIAL INQUIRY 

What types of information does your divisiWbranch currently get_ (collect orr 
receive) that relates' to performance of soldiers or use of training material* in the 
field? 

Soldier Performance Use of Ma terials 



> 
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- INFORMATION FEEDBACK NfeEDS OF CENTER/SCHOOL 

* DIRECTORATES /DEPARTMENTS 

INTERVIEW WORKSi^T 2 ^ 
Pir/Dcpt Piv/Br 



Item 



I. la what form do you get this Information? Is it useful and reliable in this form? 



2., From where do you get it? Who is your division/branch gets it? 



4 ' ** 

* 

3. What's done with the information in your division/branch? (Is its form changed? 
Is it analyzed or otherwise processed? Is it filed? Where? What conclusions and 
recommendations are developed? What reports are generated?) 



4. Where are the results of your processing of the* information sent?>»What feedbacs 
do you receive from users (of the information? If more feedback is needed; what feind?^ 



5. What document or regulation prescribes the use of this information? 
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bir/bept^ 
Item 



Div/Br- 



T 



Additional Comments ^indicate whether comments are general or keyed to questions on 
previous page) : 
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• I* : - '• * INFORMATION FEEDBACK NEEDS OF .CENTER/SCHOOL- 

DIRECTORATES /DEPARTMENTS 

: ' . . INTERVIEW WORKSHEET 3 
Dir/Dept , Dlv/Br_ ; 



> CLOSING INQUIRY 

What other types of feedback on soldier performance or use of materials do 
you heed" that you don't Currently get?* In whaf ways £sn feedback from the field 
be improved? * 



«iy 
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' APPENDIX B 
TRAINING INSPECTION REPORT 
• DATA COLLECTION FORM 
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TRAINING I^S^^TON REPORT: 



: ^r" — ■ rr 



t* \t:iori 

D ■ ■ r 



. V 



* ''iuj.ed starts. Tiine 



r >* ..Si? 



I 



'V 1 



Tu. Pilfer — • - " C 



p.. .jor.;;el Present for Training 



•p^ ,or.nel Absent *xrOm Training 



4> j ___„_.;._„,_.,^„„. - — ~ 



Ti e Evaluator Arrived, _ 



Xi. \e of Evaluator /Evaluators 



of Person Outbriefed 



* 
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BEST C9FY-.TOISBIE 



1. bid the trainer give a diagnostic examination prior to the 
instruction? ' k « 

2. If so, how many GOs , NO GOs > 

* i ■* ■» 
- * v + 
1 ; ■ 

3. What was the standard for A GO? * 



V 

.- bid. , he diagnostic? exarr. mate a- th.t>. training and the post, test? 

... * ' : • • . • v , r - > 

• j . Did, the trailer have the lesson, pl^n prepared? • i 

J 'r» ' " " " 1 - 

o\ Was the lesson plan IAW the soldiers manual task, condition « 
in4 standards? - " L 



7. Are the tasks, condi'tijjpns, and standards correct?- 

I ■ - , - - - . -J* ^> 

8 i? . Were the tasks/ conditions* ajid standards explained; 'adequately 

to the student? ';.■*?■_* ' * /* ' V 
gr. Did the trainer give the^ reason or the subject? 

10. $7as the 1 "heed to khOw H ej^S^fzed? 'K • 

11. Did the training match the training objective? 

.12 . bid the trainer display adequate knowledge of ^^*e Subject * ' ' * 

nratter? > ' ' ."'tV "V. 

; . yr \ ' . ' " ■■ . • 

13. Did the trainer give all necessary information? 

i i . Did the trainer present any unnecessary information? 



nit; t-hc t ramer ut:e understand* bio words? < 

9 " ... ' 

_ c - : i _ 

i t .'-^Sfid the trainer demonstrate how to perform the tasks * * 

v-r recti?? 



. .. . --^ • r - . . • - 

i;; *<\<arv tlic s^udeats afforded time for practical exercises? 



©id the students complete the task correctly* at least once, 

/ without coaching or assistance? • 

■ ' ■ & 

19. Was there enough time for hands on training/practice? 



* -?o^__ttas^ ijnmediate feedback given to the students after the 

vpracticef? - ■— 

i ■ ■ • . : V ■ • ' 

23*. Did the students actively participate? 

•;■ * \ 

22. Were the students enc outraged to ask questions? 

23; , Did the trainer show ihi^rtest ±ri "helping the soldier to 
learn? 1 



24. Was the training site well, organized? ^ r 

25. did the trainer arrange the training so all could see and 

hear? 



26, Wire all required support materials ^nd personnel present 
and 'on time? 6 



27. Was all support equipment operational? . 

28. Were the training materials and handouts called *or used? 

If not why not? ^ \ 

2£ were training aids introduced and explained as part of the 

+ ■ in; *;rnctioh? If hot, why hot? J 

\ - _ 

^id th« students have access to r.ftu ocdip^ent/trair.ihg aid3? 



• What was the ratio of ^instructor s/AIs per student? 



. r. it ad« qiiate for the subject? 
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best ctv ?,^yiu 



32, What was the ratio of ec^ipmeht/traihihg aids per 

student? ± — - y - 

Was it adequate for the, subject 

• - » 

33, Wore the AIs used effectively? 



34: old the Ms display adequate knowledge of the subject matter? 

< i . _ 

.».....• — ■ 

35. Was faulty performance", of either 4 the instructor or student 

identified and corrected? * 

■\ i * ' • . 

bid v the soldiers receive attest on the subject? 

37. How long did .the test occur after the completion of 
training? — — : . J 

38. Was the testes close as possible to the jotf erivirdnment? 



39. Did the training objective and the training match the test? 

; ■% * 

• _ ■" - - _ 'J -. _ _ -J _i ' 

40. Were the pass/fail standards clear arid correct? ■ 



41. Were the pass/fail standards the same for ali students? 

42. Were the specified standards (Tallied? , ' 4 



43. Were the testers the same personnel as* the instructors? 

. . : ) ... . ■ 

44 . Were the students given immediate feedback on their perform- 
ance after testing? ; :„ . . 



wi-'t- i;»v ifrtiiiiiui til -il ir/i«"t ■ ir rt «-v I « I*-*.!? 

c Sighting ' rj , Tr«ir»:r.q Araa/f^iiity 

d. Temperature 



4/^ Wag ail student training time used effectively? Ifino^ 
'Why not? ' ' ■ f ■ 

47 . Did the trainer provide a sununary of tasks', conditions, and 
standards? 



48. *Was concurrent traininq utilized darirjg;the block of 
insti-uctibri? 



49. If concurrent training was utilized, was it l^e'ctive? 

50. If test given: 
Number Tested 



1 



Number GOs 
dumber NO G0& 
Number Retested * 
Number of GOs bri Retest 



X 



US 
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FHMARKS SECTION : 



r 



> RECX)MMENDATI , ONS i 



;V ^ £FreCTI V£NE SS OF TRA I NING ; 

' • . . -\ — _-..■>' \ '■ : 

_£ ^c^pii^hed the 1 

Did not accomplish the training objective 



j ^3-j'^ ^opcqpi i?hed the training objective 
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